-- Logs begin at Sun 2022-09-18 00:10:21 UTC, end at Sun 2022-09-18 01:17:26 UTC. -- Sep 18 00:41:15.295445 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Sep 18 00:41:15.305083 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Sep 18 00:41:15.307164 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: *** Starting uWSGI 2.0.18-debian (64bit) on [Sun Sep 18 00:41:15 2022] *** Sep 18 00:41:15.307164 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Sep 18 00:41:15.307164 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Sep 18 00:41:15.307164 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: nodename: dv-cinder-master-3-iscsi Sep 18 00:41:15.307164 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: machine: x86_64 Sep 18 00:41:15.307164 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: clock source: unix Sep 18 00:41:15.307164 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: pcre jit disabled Sep 18 00:41:15.307164 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: detected number of CPU cores: 4 Sep 18 00:41:15.307164 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: current working directory: / Sep 18 00:41:15.307164 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: detected binary path: /usr/bin/uwsgi-core Sep 18 00:41:15.307683 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: your processes number limit is 47680 Sep 18 00:41:15.307683 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: your memory page size is 4096 bytes Sep 18 00:41:15.307683 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: detected max file descriptor number: 2048 Sep 18 00:41:15.307683 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: lock engine: pthread robust mutexes Sep 18 00:41:15.307683 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: thunder lock: enabled Sep 18 00:41:15.307683 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Sep 18 00:41:15.307683 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Sep 18 00:41:15.347374 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: Python main interpreter initialized at 0x55c4902d89c0 Sep 18 00:41:15.347374 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: python threads support enabled Sep 18 00:41:15.347374 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: your server socket listen backlog is limited to 100 connections Sep 18 00:41:15.347374 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: your mercy for graceful operations on workers is 90 seconds Sep 18 00:41:15.347761 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: mapped 403077 bytes (393 KB) for 2 cores Sep 18 00:41:15.347841 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: *** Operational MODE: preforking *** Sep 18 00:41:15.348090 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: *** uWSGI is running in multiple interpreter mode *** Sep 18 00:41:15.348090 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: spawned uWSGI master process (pid: 81519) Sep 18 00:41:15.348297 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Sep 18 00:41:15.349258 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: spawned uWSGI worker 1 (pid: 81529, cores: 1) Sep 18 00:41:15.349258 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: spawned uWSGI worker 2 (pid: 81530, cores: 1) Sep 18 00:41:15.349405 dv-cinder-master-3-iscsi devstack@keystone.service[81519]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Sep 18 00:41:19.656784 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 18 00:41:19.656784 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: from cryptography.utils import int_from_bytes Sep 18 00:41:19.657563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 18 00:41:19.657563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: from cryptography.utils import int_from_bytes Sep 18 00:41:19.662420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 18 00:41:19.662420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: from cryptography.utils import int_from_bytes Sep 18 00:41:19.662900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 18 00:41:19.662900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: from cryptography.utils import int_from_bytes Sep 18 00:41:19.707800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 18 00:41:19.707983 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 18 00:41:19.708163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] command line args: [] {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 18 00:41:19.708431 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 18 00:41:19.708606 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ================================================================================ {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 18 00:41:19.708860 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] admin_token = **** {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.709104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] config_dir = [] {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.709290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.709469 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] config_source = [] {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.709646 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] debug = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.709842 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.710338 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.710427 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.710649 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.710895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.711191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] list_limit = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.711296 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] log_config_append = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.711367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 18 00:41:19.711488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.711579 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 18 00:41:19.711646 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] log_dir = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.711708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] command line args: [] {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 18 00:41:19.711841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] log_file = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.711985 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 18 00:41:19.712082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.712205 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ================================================================================ {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 18 00:41:19.712278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.712390 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] admin_token = **** {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.712494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.712569 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] config_dir = [] {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.712739 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.712826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.712920 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.712996 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] config_source = [] {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.713120 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.713194 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] debug = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.713307 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.713449 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.713534 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.713653 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.713727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.713844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.713917 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.713985 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.714171 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.714250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.714407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] list_limit = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.714505 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.714613 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] log_config_append = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.714688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.714794 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.714921 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.714996 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] log_dir = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.715139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] log_file = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.715214 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.715377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.715632 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.715712 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.42/identity {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.715835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] publish_errors = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.715905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.716065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.716149 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.716283 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.716366 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.716457 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.716534 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.716700 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.716781 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.716933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.717007 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.717164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] standard_threads = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.717239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.717393 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.717467 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.717629 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.717700 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.717799 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.717871 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.717972 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] use_journal = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.718130 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.718209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] use_json = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.718330 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] use_stderr = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.718411 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.718525 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] use_syslog = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.718671 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.718757 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.718997 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.719170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.42/identity {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.719256 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.719358 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] publish_errors = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.719480 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.719597 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.719735 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.719835 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.719952 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.720027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.720190 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.720264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.720437 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.720513 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.720672 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.720753 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] standard_threads = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.720918 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.720993 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.721163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.721237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.721343 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.721416 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.721521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] use_journal = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.721631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.721713 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] use_json = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.721864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] use_stderr = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.721943 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.722042 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] use_syslog = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.722118 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.722278 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 18 00:41:19.722359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.722562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.722640 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.722804 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.722879 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.723060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.723144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.723302 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.723382 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.723545 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.723624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.723780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.723863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.724030 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.724104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.724271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.724346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.724522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.724607 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.724766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.724841 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.724996 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.725070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.725224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.725372 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.725450 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.725518 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.725672 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.725748 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.725919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.726001 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.726162 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.726236 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.726389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.726483 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.726640 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.726715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.726882 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.726957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.727125 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.727200 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.727359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.727435 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.727603 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.727676 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.727833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.727913 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.728067 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.728143 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.728298 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.728373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.728526 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.728599 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.728752 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.728833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.728980 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.729061 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.729207 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.729286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.729442 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.729518 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.729687 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.729761 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.729915 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.729989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.730149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.730223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.730376 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.730449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.730634 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.730708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.730851 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.730929 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.731100 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.731174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.731328 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.731402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.731563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.731639 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.731797 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.731871 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.732030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.732103 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.732263 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.732338 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.732495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.732573 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.732719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.732792 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.732966 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.733077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.733198 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.733289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.733442 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.733523 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.734202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.734294 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.734449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.734613 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.734696 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.734766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.734934 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.735010 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.735172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.735267 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.735409 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.735486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.735650 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.735725 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.735893 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.735975 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.736134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.736217 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.736374 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.736472 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.736608 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.736683 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.736840 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.736914 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.737072 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.737154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.737319 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.737433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.737556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.737630 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.737790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.737873 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.738030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.738106 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.738268 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.738343 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.738529 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.738624 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.738763 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.738838 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.738997 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.739073 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.739231 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.739313 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.739465 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.739540 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.739699 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.739773 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.739959 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.740042 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.740198 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.740273 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.740436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.740517 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.740673 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.740756 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.740909 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.740984 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.741148 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.741223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.741384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.741466 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.741619 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.741689 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.741851 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.741927 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.742080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.742163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.742313 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.742413 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.742563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.742644 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.742796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.742966 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.743042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.743113 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.743255 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.743339 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.743489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.743575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.743734 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.743813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.743969 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.744046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.744223 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.744315 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.744465 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.744555 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.744709 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.744785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.744939 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] auth.external = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.745017 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.745172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.745251 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.745415 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.745490 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.745652 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.745735 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.745888 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.745972 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] auth.password = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.746119 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] auth.token = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.746195 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.746358 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.746433 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.746615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.746703 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.746861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.746947 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.747090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.747182 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.747337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.747433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.747575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.747653 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.747804 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.747880 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.748039 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] credential.caching = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.748117 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.748283 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.748363 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.748529 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.748607 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] auth.external = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.748763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.748849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.749004 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.749092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.749247 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.749325 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.750063 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.750145 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] auth.password = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.750304 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] auth.token = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.750382 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.750575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.750657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.750812 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.750890 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.751062 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.751142 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.751300 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.751378 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.751551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.751637 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.751805 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.751885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.752042 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.752126 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.752303 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] credential.caching = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.752383 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.752519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] federation.caching = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.752598 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.752758 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.752836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.753010 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.753087 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.753256 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.753336 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.753486 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.753570 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.753731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.753810 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.753966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.754046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.754218 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.754298 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.754469 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.754551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.754726 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.754807 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.754963 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.755041 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.755208 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.755294 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.755443 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] identity.caching = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.755522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.755689 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.755769 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.755935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.756025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.756171 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.756249 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] federation.caching = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.756407 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.756491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.756653 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.756740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.756884 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.756963 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.757119 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.757197 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.757367 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.757534 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.757611 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.757681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.757833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.757925 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.758085 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.758196 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.758291 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.758374 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.758556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.758639 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.758804 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.758896 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.759046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] identity.caching = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.759124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.759300 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.759378 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.759547 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.759633 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.759787 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.759865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.760030 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.760108 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.760265 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.760350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.760497 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.760577 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.760733 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.760812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.760964 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.761050 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.761199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.761277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.761436 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.761515 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.761671 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.761756 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.761909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.761993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.762135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.762212 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.762382 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.762481 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.762653 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.762732 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.762894 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.762982 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.763141 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.763227 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.763386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.763464 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.763617 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.763695 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.763855 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.763943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.764084 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.764162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.764320 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.764398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.764554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.764642 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.764794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.764871 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.765082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.765159 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.765331 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.765416 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.765654 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.765746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.765818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.765968 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.766067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.766203 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.766307 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.766435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.766578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.766708 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.766797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.766958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.767036 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.767197 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.767276 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.767436 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.767521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.767673 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.767751 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.767906 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.user = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.767983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.768147 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.768233 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.768406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.768484 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.768637 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.768715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.768879 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.768963 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.769125 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.769206 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.769372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.769450 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.769606 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.769690 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.769834 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.769912 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.770069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.770147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.770303 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.770389 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.770571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.770652 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.770802 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.770880 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.771043 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.771131 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.771282 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.user = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.771361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.771525 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.771602 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.771770 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.771856 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.772009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.772086 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.772235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.772312 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.772481 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.772567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.772716 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.772795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.772957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.773035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.773190 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.773274 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.773422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.773499 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.773661 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.773737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.773902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.773988 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.774136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.774214 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.774384 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.774506 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.774636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.774726 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.774880 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.774959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.775117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.775195 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.775353 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.775439 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.775590 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.775711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.775837 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.775916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.776078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.776164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.776307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.776390 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.776543 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.776622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.776782 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.776867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] resource.caching = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.777028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.777106 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.777276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.777402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.777513 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.777651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.777761 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.777914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.778012 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.778152 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.778250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.778406 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.778504 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.778676 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.778754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.778920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.779006 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.779167 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.779300 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] role.caching = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.779429 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.779556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] role.driver = sql {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.779684 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.779810 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.779957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.780086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.780213 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.780342 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.780469 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.780698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] resource.caching = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.780698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.780860 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.780933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.781003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.781136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.781224 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.781383 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.781462 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.781631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.781756 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.781865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.781952 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.782110 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.782189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.782352 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.782429 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.782620 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.782711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.782866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] role.caching = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.782945 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.783121 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] role.driver = sql {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.783301 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.783410 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.783533 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.783615 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.783783 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.783861 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.784033 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.784110 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.784280 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.784358 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.784528 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.784605 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.784764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.784841 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.785019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.785098 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.785270 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.785347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.785507 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.785584 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.785738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.785817 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.785969 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.786045 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.786199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.786276 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.786448 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.786554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.786723 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.786903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.786981 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.787146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.787226 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.787403 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.787480 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.787633 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.787709 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.787870 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] token.caching = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.787948 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.788105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.788182 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.788351 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.788428 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.788587 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.788665 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.788840 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.788917 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.789086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.789161 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.789321 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.789399 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.789555 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.789631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.789793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.789869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.790037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.790113 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.790271 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.790354 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.790530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.790612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.790766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.790845 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.791010 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.791087 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.791244 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.791321 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.791487 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] token.caching = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.791565 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.791731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.791815 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.791982 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.792058 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.792240 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.792429 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] database.connection = **** {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.792506 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.792575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.792689 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.792766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.792930 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.793016 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.793170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.793247 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.793406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.793483 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.793642 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.793726 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.793878 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.793955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.794113 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.794192 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.794355 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.794438 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.794612 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.794690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.794856 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.794932 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.795090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.795178 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.795335 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.795423 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.795569 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.795646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.795812 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.795914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] database.connection = **** {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.796042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.796670 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.796797 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.796934 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.797034 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.797179 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.797256 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81530) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 18 00:41:19.797425 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.797665 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.797907 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.798148 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.798388 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.798663 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.798905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.799155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.799650 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.799901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.800142 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.800394 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.800629 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.800876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.801118 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 18 00:41:19.801290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81529) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 18 00:41:19.826583 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:19.826809 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:19.827036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:19.830558 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:19.830795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:19.831026 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:19.835895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:19.839442 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:19.864379 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:19.864606 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:19.866028 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:19.866266 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:19.918362 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:19.918899 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:19.967203 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:19.967426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:19.968838 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:19.969076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.123595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.123824 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.123965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.124054 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.124202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.124774 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.124948 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.125030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.125165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.125256 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.125403 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.125603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.147955 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.148178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.148261 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.148397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.148481 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.148614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.148696 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.148828 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.148918 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.149039 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.149121 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.149249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.149329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.149534 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.161002 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.161630 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.173847 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.174079 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.174303 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.174733 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.174820 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.174972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.175052 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.175200 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.175294 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.175409 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81530) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.175489 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.175643 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.175863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.176082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81529) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 18 00:41:20.193244 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.193450 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 18 00:41:20.193798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.193877 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.193998 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 18 00:41:20.194100 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 18 00:41:20.194269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.194349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.194549 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.194631 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 18 00:41:20.194808 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.194946 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.195068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.195190 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.195350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.195613 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.204691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.204967 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.205239 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.205524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.207003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.207380 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.207661 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.207995 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.212708 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81530) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.215605 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81530) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.215889 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.216014 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81529) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.216170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.216450 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.216732 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.216993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.217259 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.217529 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.217890 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.218268 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.218576 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.219183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81529) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.219475 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.219766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.220056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.220348 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.220616 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.220891 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.221171 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.221449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.221726 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.221998 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.241426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.242908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81530) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.243193 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.245096 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.246610 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81529) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.246900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.247821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81530) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.248105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.248385 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.252213 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81529) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.252500 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.252786 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.255244 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81530) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.255528 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.259169 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81529) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.259456 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.261262 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81530) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.261559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.261849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.262136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.262416 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.262718 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.262993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.263265 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.265310 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81529) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.265607 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.265895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.266187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.266482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.266770 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.267046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.267318 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.278692 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.278985 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.283201 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81530) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.284022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.284330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.286504 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.286805 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.288657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81529) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.291022 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81530) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.292748 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.293048 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.294200 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81530) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.297992 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.298163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81529) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.298290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.298592 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.298874 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.299148 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.299687 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.302270 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81529) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.306286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.306649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.306943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.307258 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.307602 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.307990 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.315928 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81530) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.316219 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.316508 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.316747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.323694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.325220 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.326739 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81529) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.326825 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.327041 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.327121 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.327324 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.327633 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.329752 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81530) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.330038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.330326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.330638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.331031 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.334505 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.336059 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.337596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.337876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.340677 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81529) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.341190 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.341673 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.342017 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81530) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.342098 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.342302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.342611 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.342740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.342892 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.343294 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.343587 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.343864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.353894 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81529) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.354188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.354501 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.354787 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.355178 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.355453 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.355742 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.359583 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81530) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.362467 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81530) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.365322 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.366767 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.368224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81530) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.368503 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.368898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.371596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81529) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.375177 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81529) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.375772 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.377241 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81530) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.378033 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.379609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.380128 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.380520 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.380804 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.381080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.381179 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81529) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.381367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.381771 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.388182 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.388802 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81530) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.389093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.389371 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.389653 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81529) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.393106 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.393506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.393796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.394075 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.396739 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81530) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.397019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.397293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.397572 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.397840 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.398109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.398377 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.398683 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.398950 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.399225 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.399542 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.399851 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.400118 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.400396 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81530) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.401905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81529) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.402201 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.402509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.409976 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81529) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 18 00:41:20.410264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.410576 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.410856 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.411130 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.411403 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.411679 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.411966 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.412240 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.412529 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.412805 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.413068 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.413335 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.413609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81529) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 18 00:41:20.434901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55c4902d89c0 pid: 81530 (default app) Sep 18 00:41:20.438215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1d98f6-2498-4539-9797-28f0b2b5b445 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:20.438518 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1d98f6-2498-4539-9797-28f0b2b5b445 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:20.438799 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1d98f6-2498-4539-9797-28f0b2b5b445 None None] PATH_INFO: `/v3` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:20.442893 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1/1] 10.222.0.42 () {50 vars in 791 bytes} [Sun Sep 18 00:41:20 2022] GET /identity/v3/ => generated 255 bytes in 7 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:20.447816 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55c4902d89c0 pid: 81529 (default app) Sep 18 00:41:27.224263 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13127c7-ef38-466d-b9c9-488d2e7825ba None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:27.224776 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13127c7-ef38-466d-b9c9-488d2e7825ba None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:27.224776 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13127c7-ef38-466d-b9c9-488d2e7825ba None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:27.226093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 2/2] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:41:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:41:27.234416 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecb8d1a-6955-4599-984c-f02844676b09 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:27.234881 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecb8d1a-6955-4599-984c-f02844676b09 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:27.235019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecb8d1a-6955-4599-984c-f02844676b09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:27.305329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: INFO dbcounter [None req-1ecb8d1a-6955-4599-984c-f02844676b09 None None] Registered counter for database keystone Sep 18 00:41:27.324540 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG oslo_db.sqlalchemy.engines [None req-1ecb8d1a-6955-4599-984c-f02844676b09 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81529) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 18 00:41:27.331377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writer thread running {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 18 00:41:27.525699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG passlib.handlers.bcrypt [None req-1ecb8d1a-6955-4599-984c-f02844676b09 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81529) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 18 00:41:27.526109 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG passlib.handlers.bcrypt [None req-1ecb8d1a-6955-4599-984c-f02844676b09 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81529) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 18 00:41:27.586647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-1ecb8d1a-6955-4599-984c-f02844676b09 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:41:27.588296 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-1ecb8d1a-6955-4599-984c-f02844676b09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:41:27.731539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1/3] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:41:27 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 501 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Sep 18 00:41:28.028594 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13127c7-ef38-466d-b9c9-488d2e7825ba None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:28.098753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: INFO dbcounter [None req-c13127c7-ef38-466d-b9c9-488d2e7825ba None None] Registered counter for database keystone Sep 18 00:41:28.120600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG oslo_db.sqlalchemy.engines [None req-c13127c7-ef38-466d-b9c9-488d2e7825ba None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81530) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 18 00:41:28.126815 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writer thread running {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 18 00:41:28.246167 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13127c7-ef38-466d-b9c9-488d2e7825ba None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:28.251156 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f80d71b-d34d-403b-bcea-67dc502a3998 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:28.251784 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f80d71b-d34d-403b-bcea-67dc502a3998 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:28.252069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f80d71b-d34d-403b-bcea-67dc502a3998 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:28.252345 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f80d71b-d34d-403b-bcea-67dc502a3998 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:28.260629 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f80d71b-d34d-403b-bcea-67dc502a3998 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:28.314497 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: WARNING keystone.common.rbac_enforcer.enforcer [None req-2f80d71b-d34d-403b-bcea-67dc502a3998 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 18 00:41:28.314859 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f80d71b-d34d-403b-bcea-67dc502a3998 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:28.324922 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 3/4] 10.222.0.42 () {58 vars in 1324 bytes} [Sun Sep 18 00:41:28 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 18 00:41:28.330955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecb8d1a-6955-4599-984c-f02844676b09 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:28.346151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecb8d1a-6955-4599-984c-f02844676b09 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:28.350689 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd2e7ff-fa5e-40ce-a84e-5b25aeec677f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:28.351864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd2e7ff-fa5e-40ce-a84e-5b25aeec677f admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:28.352149 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd2e7ff-fa5e-40ce-a84e-5b25aeec677f admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:28.352423 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd2e7ff-fa5e-40ce-a84e-5b25aeec677f admin admin] PATH_INFO: `/v3/projects/2e41e0644d514666af8ca7ed05ebaec8` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:28.353929 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd2e7ff-fa5e-40ce-a84e-5b25aeec677f admin admin] RBAC: Authorizing `identity:get_project(project_id=2e41e0644d514666af8ca7ed05ebaec8)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:28.409415 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: WARNING keystone.common.rbac_enforcer.enforcer [None req-9dd2e7ff-fa5e-40ce-a84e-5b25aeec677f admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 18 00:41:28.409762 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd2e7ff-fa5e-40ce-a84e-5b25aeec677f admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:28.414499 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 2/5] 10.222.0.42 () {56 vars in 1207 bytes} [Sun Sep 18 00:41:28 2022] GET /identity/v3/projects/2e41e0644d514666af8ca7ed05ebaec8 => generated 344 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:30.540166 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd062b0-7762-435e-bb36-ddce2f985a2b None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:30.540736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd062b0-7762-435e-bb36-ddce2f985a2b None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:30.540736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd062b0-7762-435e-bb36-ddce2f985a2b None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:30.544813 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 4/6] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:41:30 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:41:30.551996 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb880740-248c-43b4-a64d-e9ad9d6d4f63 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:30.552453 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb880740-248c-43b4-a64d-e9ad9d6d4f63 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:30.552544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb880740-248c-43b4-a64d-e9ad9d6d4f63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:30.572113 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-cb880740-248c-43b4-a64d-e9ad9d6d4f63 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:41:30.573633 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-cb880740-248c-43b4-a64d-e9ad9d6d4f63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:41:30.581433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 3/7] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:41:30 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 18 00:41:30.850427 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd062b0-7762-435e-bb36-ddce2f985a2b None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:30.863095 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd062b0-7762-435e-bb36-ddce2f985a2b None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:30.867355 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0dbecd-0a1a-46c9-93a8-2702f4540068 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:30.867967 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0dbecd-0a1a-46c9-93a8-2702f4540068 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:30.868246 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0dbecd-0a1a-46c9-93a8-2702f4540068 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:30.868518 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0dbecd-0a1a-46c9-93a8-2702f4540068 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:30.876682 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0dbecd-0a1a-46c9-93a8-2702f4540068 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:30.878447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0dbecd-0a1a-46c9-93a8-2702f4540068 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:30.887408 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 5/8] 10.222.0.42 () {58 vars in 1324 bytes} [Sun Sep 18 00:41:30 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 18 00:41:30.893993 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb880740-248c-43b4-a64d-e9ad9d6d4f63 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:30.903131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb880740-248c-43b4-a64d-e9ad9d6d4f63 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:30.908123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3faf90-6c09-493f-a1f0-7abc464f4bbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:30.908747 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3faf90-6c09-493f-a1f0-7abc464f4bbf admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:30.909026 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3faf90-6c09-493f-a1f0-7abc464f4bbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:30.909301 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3faf90-6c09-493f-a1f0-7abc464f4bbf admin admin] PATH_INFO: `/v3/users/6f5558a706da458791737ed054e51874` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:30.910772 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3faf90-6c09-493f-a1f0-7abc464f4bbf admin admin] RBAC: Authorizing `identity:get_user(user_id=6f5558a706da458791737ed054e51874)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:30.912595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3faf90-6c09-493f-a1f0-7abc464f4bbf admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:30.914132 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 4/9] 10.222.0.42 () {56 vars in 1198 bytes} [Sun Sep 18 00:41:30 2022] GET /identity/v3/users/6f5558a706da458791737ed054e51874 => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:33.136768 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02bb4b0-0e8b-463b-af6b-82ab41a7fe10 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:33.137661 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02bb4b0-0e8b-463b-af6b-82ab41a7fe10 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:33.137941 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02bb4b0-0e8b-463b-af6b-82ab41a7fe10 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:33.139832 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 6/10] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:41:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:41:33.145930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff4af24-fda7-4c6d-a487-a30abdd45d5a None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:33.146385 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff4af24-fda7-4c6d-a487-a30abdd45d5a None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:33.146728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff4af24-fda7-4c6d-a487-a30abdd45d5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:33.168750 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-7ff4af24-fda7-4c6d-a487-a30abdd45d5a None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:41:33.169960 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-7ff4af24-fda7-4c6d-a487-a30abdd45d5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:41:33.217109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3af511-1ab2-4496-8832-24d02f21e053 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:33.217388 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3af511-1ab2-4496-8832-24d02f21e053 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:33.217664 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3af511-1ab2-4496-8832-24d02f21e053 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:33.219023 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 7/11] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:41:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:41:33.224908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e58e26f-ec62-4cf5-bbfa-f2a817e82523 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:33.225187 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e58e26f-ec62-4cf5-bbfa-f2a817e82523 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:33.225462 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e58e26f-ec62-4cf5-bbfa-f2a817e82523 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:33.256324 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 5/12] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:41:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:41:33.269423 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG passlib.handlers.bcrypt [None req-4e58e26f-ec62-4cf5-bbfa-f2a817e82523 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81530) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 18 00:41:33.269825 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG passlib.handlers.bcrypt [None req-4e58e26f-ec62-4cf5-bbfa-f2a817e82523 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81530) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 18 00:41:33.319753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-4e58e26f-ec62-4cf5-bbfa-f2a817e82523 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:41:33.321381 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-4e58e26f-ec62-4cf5-bbfa-f2a817e82523 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:41:33.430020 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 8/13] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:41:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:41:33.535228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff4af24-fda7-4c6d-a487-a30abdd45d5a None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:33.564039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:33.661702 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff4af24-fda7-4c6d-a487-a30abdd45d5a None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:33.700234 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f092705e-fe8e-460e-a279-092da1083fc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:33.700796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f092705e-fe8e-460e-a279-092da1083fc1 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:33.701074 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f092705e-fe8e-460e-a279-092da1083fc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:33.701344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f092705e-fe8e-460e-a279-092da1083fc1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:33.712897 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e58e26f-ec62-4cf5-bbfa-f2a817e82523 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:33.716187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f092705e-fe8e-460e-a279-092da1083fc1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:41:33.725078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-f092705e-fe8e-460e-a279-092da1083fc1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:41:33.725078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:41:33.725078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:41:33.725078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:41:33.725078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:41:33.725078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:41:33.725078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:41:33.725078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:41:33.725078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:41:33.725078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:41:33.725078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:41:33.725078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:41:33.725078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:41:33.725078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:41:33.725078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:41:33.725078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:41:33.726123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:41:33.726123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:41:33.726123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:41:33.726123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:41:33.726123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:41:33.726123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:41:33.726123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:41:33.726123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:41:33.726123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:41:33.726123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:41:33.726123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:41:33.726123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:41:33.726123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:41:33.726123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:41:33.726123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:41:33.726123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:41:33.726994 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:41:33.726994 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:41:33.726994 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:41:33.726994 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:41:33.726994 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:41:33.726994 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:41:33.726994 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:41:33.726994 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:41:33.726994 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:41:33.726994 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:41:33.726994 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:41:33.726994 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:41:33.731289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 6/14] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:41:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:41:33.733567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f092705e-fe8e-460e-a279-092da1083fc1 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:33.825113 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e58e26f-ec62-4cf5-bbfa-f2a817e82523 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:33.828091 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:33.831686 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:33.859548 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f092705e-fe8e-460e-a279-092da1083fc1 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:33.870070 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e93060-cb5b-41d9-a40c-b3519358b7d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:33.870988 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e93060-cb5b-41d9-a40c-b3519358b7d3 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:33.871279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e93060-cb5b-41d9-a40c-b3519358b7d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:33.871556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e93060-cb5b-41d9-a40c-b3519358b7d3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:33.901807 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d98c497-4ba4-4c48-a5de-d36ebe50cf74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:33.902440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d98c497-4ba4-4c48-a5de-d36ebe50cf74 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:33.902745 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d98c497-4ba4-4c48-a5de-d36ebe50cf74 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:33.903019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d98c497-4ba4-4c48-a5de-d36ebe50cf74 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:33.903737 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d98c497-4ba4-4c48-a5de-d36ebe50cf74 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:41:33.904050 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d98c497-4ba4-4c48-a5de-d36ebe50cf74 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:33.906049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d98c497-4ba4-4c48-a5de-d36ebe50cf74 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:33.918381 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 7/15] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:41:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:41:33.924885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d98c497-4ba4-4c48-a5de-d36ebe50cf74 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:33.988799 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e93060-cb5b-41d9-a40c-b3519358b7d3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:33.990829 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e93060-cb5b-41d9-a40c-b3519358b7d3 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:34.050452 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d98c497-4ba4-4c48-a5de-d36ebe50cf74 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:34.103711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67bec67e-347b-4cd9-b19f-d2c3dedb6407 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:34.104512 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bec67e-347b-4cd9-b19f-d2c3dedb6407 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:34.104870 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bec67e-347b-4cd9-b19f-d2c3dedb6407 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:34.105253 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bec67e-347b-4cd9-b19f-d2c3dedb6407 None admin] PATH_INFO: `/v3/users/6f5558a706da458791737ed054e51874` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:34.106640 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67bec67e-347b-4cd9-b19f-d2c3dedb6407 None admin] RBAC: Authorizing `identity:get_user(user_id=6f5558a706da458791737ed054e51874)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:34.108557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67bec67e-347b-4cd9-b19f-d2c3dedb6407 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:34.110395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 8/16] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:41:33 2022] GET /identity/v3/users/6f5558a706da458791737ed054e51874 => generated 247 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:41:34.116978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67bec67e-347b-4cd9-b19f-d2c3dedb6407 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:34.127240 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 9/17] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:41:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 18 00:41:34.135872 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e93060-cb5b-41d9-a40c-b3519358b7d3 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:34.137495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:34.140562 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:34.255384 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67bec67e-347b-4cd9-b19f-d2c3dedb6407 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:34.277543 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e93060-cb5b-41d9-a40c-b3519358b7d3 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:34.305702 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6394fce0-17ef-4707-b8cb-74794d405995 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:34.306561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6394fce0-17ef-4707-b8cb-74794d405995 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:34.306936 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6394fce0-17ef-4707-b8cb-74794d405995 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:34.307293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6394fce0-17ef-4707-b8cb-74794d405995 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:34.308858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6394fce0-17ef-4707-b8cb-74794d405995 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:34.310512 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6394fce0-17ef-4707-b8cb-74794d405995 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:34.313140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 9/18] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:41:34 2022] GET /identity/v3/domains/default => generated 199 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:41:34.319738 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6394fce0-17ef-4707-b8cb-74794d405995 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:34.329414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49c93b2-b22a-4c5a-8788-2646f1ce0bc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:34.330485 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49c93b2-b22a-4c5a-8788-2646f1ce0bc8 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:34.330867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49c93b2-b22a-4c5a-8788-2646f1ce0bc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:34.331230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49c93b2-b22a-4c5a-8788-2646f1ce0bc8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:34.353382 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49c93b2-b22a-4c5a-8788-2646f1ce0bc8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:34.355146 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49c93b2-b22a-4c5a-8788-2646f1ce0bc8 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:34.382071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-a49c93b2-b22a-4c5a-8788-2646f1ce0bc8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:41:34.382071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:41:34.382071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:41:34.382071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:41:34.382071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:41:34.382071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:41:34.382071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:41:34.382071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:41:34.382071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:41:34.382071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:41:34.382071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:41:34.382071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:41:34.382071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:41:34.382071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:41:34.382071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:41:34.382071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:41:34.384093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:41:34.384093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:41:34.384093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:41:34.384093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:41:34.384093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:41:34.384093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:41:34.384093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:41:34.384093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:41:34.384093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:41:34.384093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:41:34.384093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:41:34.384093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:41:34.384093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:41:34.384093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:41:34.384093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:41:34.384093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:41:34.384093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:41:34.385625 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:41:34.385625 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:41:34.385625 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:41:34.385625 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 10/19] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:41:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:41:34.393375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49c93b2-b22a-4c5a-8788-2646f1ce0bc8 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:34.421566 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:34.448245 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6394fce0-17ef-4707-b8cb-74794d405995 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:34.466292 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:34.495490 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0070baec-dbdd-4daf-94d6-eded800158b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:34.496267 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0070baec-dbdd-4daf-94d6-eded800158b7 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:34.496619 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0070baec-dbdd-4daf-94d6-eded800158b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:34.496955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0070baec-dbdd-4daf-94d6-eded800158b7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:34.497834 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0070baec-dbdd-4daf-94d6-eded800158b7 None admin] RBAC: Adding query filter params (role.id=1e04c9fea65244d28e1e59fc4c64f32f, scope.domain.id=default, user.id=6f5558a706da458791737ed054e51874) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:41:34.498260 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0070baec-dbdd-4daf-94d6-eded800158b7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:34.499925 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0070baec-dbdd-4daf-94d6-eded800158b7 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:34.507859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 10/20] 10.222.0.42 () {56 vars in 1322 bytes} [Sun Sep 18 00:41:34 2022] GET /identity/v3/role_assignments?user.id=6f5558a706da458791737ed054e51874&scope.domain.id=default&role.id=1e04c9fea65244d28e1e59fc4c64f32f => generated 233 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:41:34.520415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49c93b2-b22a-4c5a-8788-2646f1ce0bc8 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:34.563408 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ea1c4c-4355-4f1c-9b52-aac0e82f459e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:34.564033 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ea1c4c-4355-4f1c-9b52-aac0e82f459e None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:34.564315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ea1c4c-4355-4f1c-9b52-aac0e82f459e None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:34.564592 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ea1c4c-4355-4f1c-9b52-aac0e82f459e None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:34.565267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ea1c4c-4355-4f1c-9b52-aac0e82f459e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:41:34.565571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ea1c4c-4355-4f1c-9b52-aac0e82f459e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:34.567092 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ea1c4c-4355-4f1c-9b52-aac0e82f459e None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:34.587930 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 11/21] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:41:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:41:37.347124 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a40b77-2fad-43da-a1be-ab732f97ae38 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:37.348650 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a40b77-2fad-43da-a1be-ab732f97ae38 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:37.349420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a40b77-2fad-43da-a1be-ab732f97ae38 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:37.352550 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 11/22] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:41:37 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:41:37.362374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14910fe-3c2a-4620-b424-c355ef0ec665 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:37.363886 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14910fe-3c2a-4620-b424-c355ef0ec665 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:37.364616 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14910fe-3c2a-4620-b424-c355ef0ec665 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:37.404279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-b14910fe-3c2a-4620-b424-c355ef0ec665 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:41:37.407330 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-b14910fe-3c2a-4620-b424-c355ef0ec665 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:41:37.593634 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 12/23] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:41:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:41:38.227025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956b8edc-951c-4359-9d02-978c229ad8cc None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:38.228030 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956b8edc-951c-4359-9d02-978c229ad8cc None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:38.228819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956b8edc-951c-4359-9d02-978c229ad8cc None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:38.231942 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 12/24] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:41:38 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:41:38.241550 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b746b49-2e2d-45f0-ac74-d3c8dbd9a51f None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:38.242504 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b746b49-2e2d-45f0-ac74-d3c8dbd9a51f None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:38.243258 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b746b49-2e2d-45f0-ac74-d3c8dbd9a51f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:38.283178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-0b746b49-2e2d-45f0-ac74-d3c8dbd9a51f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:41:38.285486 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-0b746b49-2e2d-45f0-ac74-d3c8dbd9a51f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:41:38.402515 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:38.457912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 13/25] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:41:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:41:38.577633 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956b8edc-951c-4359-9d02-978c229ad8cc None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:38.756926 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a25b88-13fc-4e4e-9e51-08131e110149 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:38.757878 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a25b88-13fc-4e4e-9e51-08131e110149 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:38.758656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a25b88-13fc-4e4e-9e51-08131e110149 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:38.761332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 14/26] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:41:38 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:41:38.772747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddb1b0e-de4f-45af-8207-ddbbbdacdcf5 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:38.773615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddb1b0e-de4f-45af-8207-ddbbbdacdcf5 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:38.774335 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddb1b0e-de4f-45af-8207-ddbbbdacdcf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:38.808605 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956b8edc-951c-4359-9d02-978c229ad8cc None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:38.844836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-eddb1b0e-de4f-45af-8207-ddbbbdacdcf5 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:41:38.846892 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-eddb1b0e-de4f-45af-8207-ddbbbdacdcf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:41:38.892866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7bd27a-9495-4fd1-816e-935906add4f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:38.893536 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7bd27a-9495-4fd1-816e-935906add4f8 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:38.893832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7bd27a-9495-4fd1-816e-935906add4f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:38.894081 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7bd27a-9495-4fd1-816e-935906add4f8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:38.918906 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7bd27a-9495-4fd1-816e-935906add4f8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:38.920499 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7bd27a-9495-4fd1-816e-935906add4f8 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:38.928753 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:38.949851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-9e7bd27a-9495-4fd1-816e-935906add4f8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:41:38.949851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:41:38.949851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:41:38.949851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:41:38.949851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:41:38.949851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:41:38.949851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:41:38.949851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:41:38.949851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:41:38.949851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:41:38.949851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:41:38.949851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:41:38.949851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:41:38.949851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:41:38.949851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:41:38.949851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:41:38.951510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:41:38.951510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:41:38.951510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:41:38.951510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:41:38.951510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:41:38.951510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:41:38.951510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:41:38.951510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:41:38.951510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:41:38.951510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:41:38.951510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:41:38.951510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:41:38.951510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:41:38.951510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:41:38.951510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:41:38.951510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:41:38.951510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:41:38.953062 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:41:38.953062 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:41:38.953062 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:41:38.953062 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 13/27] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:41:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 376 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:41:38.960432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7bd27a-9495-4fd1-816e-935906add4f8 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:39.124797 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 15/28] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:41:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 354 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:41:39.194328 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7bd27a-9495-4fd1-816e-935906add4f8 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:39.210497 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddb1b0e-de4f-45af-8207-ddbbbdacdcf5 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:39.267336 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7011caf-7991-4484-8879-402f6b739eed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:39.268042 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7011caf-7991-4484-8879-402f6b739eed None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:39.268394 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7011caf-7991-4484-8879-402f6b739eed None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:39.268677 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7011caf-7991-4484-8879-402f6b739eed None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:39.269371 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7011caf-7991-4484-8879-402f6b739eed None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:41:39.269680 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7011caf-7991-4484-8879-402f6b739eed None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:39.271205 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7011caf-7991-4484-8879-402f6b739eed None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:39.316131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 14/29] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:41:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 356 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:41:39.317208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4732872-6f17-40b3-9c85-256cd02a63d0 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:39.317493 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4732872-6f17-40b3-9c85-256cd02a63d0 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:39.317771 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4732872-6f17-40b3-9c85-256cd02a63d0 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:39.320611 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 15/30] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:41:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:41:39.321647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4732872-6f17-40b3-9c85-256cd02a63d0 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:39.433859 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddb1b0e-de4f-45af-8207-ddbbbdacdcf5 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:39.444259 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:39.450864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:39.512326 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4732872-6f17-40b3-9c85-256cd02a63d0 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:39.530912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fa0016-e485-4e65-95bc-7b931d86717b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:39.531585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fa0016-e485-4e65-95bc-7b931d86717b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:39.531864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fa0016-e485-4e65-95bc-7b931d86717b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:39.532133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fa0016-e485-4e65-95bc-7b931d86717b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:39.546274 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fa0016-e485-4e65-95bc-7b931d86717b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:41:39.553403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-c5fa0016-e485-4e65-95bc-7b931d86717b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:41:39.553403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:41:39.553403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:41:39.553403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:41:39.553403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:41:39.553403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:41:39.553403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:41:39.553403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:41:39.553403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:41:39.553403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:41:39.553403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:41:39.553403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:41:39.553403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:41:39.553403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:41:39.553403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:41:39.553403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:41:39.554439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:41:39.554439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:41:39.554439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:41:39.554439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:41:39.554439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:41:39.554439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:41:39.554439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:41:39.554439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:41:39.554439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:41:39.554439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:41:39.554439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:41:39.554439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:41:39.554439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:41:39.554439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:41:39.554439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:41:39.554439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:41:39.555293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:41:39.555293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:41:39.555293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:41:39.555293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:41:39.555293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:41:39.555293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:41:39.555293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:41:39.555293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:41:39.555293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:41:39.555293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:41:39.555293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:41:39.555293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:41:39.555293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 16/31] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:41:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 345 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:41:39.560714 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47964079-d462-49c1-8053-9fe1db625e6f None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:39.560714 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47964079-d462-49c1-8053-9fe1db625e6f None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:39.560714 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47964079-d462-49c1-8053-9fe1db625e6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:39.573109 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3e369f-475b-4509-9aa9-48cb45d563eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:39.574152 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3e369f-475b-4509-9aa9-48cb45d563eb None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:39.574630 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3e369f-475b-4509-9aa9-48cb45d563eb None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:39.575063 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3e369f-475b-4509-9aa9-48cb45d563eb None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:39.575925 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3e369f-475b-4509-9aa9-48cb45d563eb None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:39.577793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3e369f-475b-4509-9aa9-48cb45d563eb None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:39.580032 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-47964079-d462-49c1-8053-9fe1db625e6f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:41:39.581245 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-47964079-d462-49c1-8053-9fe1db625e6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:41:39.671294 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 17/32] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:41:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:41:39.673190 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47964079-d462-49c1-8053-9fe1db625e6f None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:39.736026 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 16/33] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 18 00:41:39 2022] POST /identity/v3/projects => generated 301 bytes in 409 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:41:39.736026 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3e369f-475b-4509-9aa9-48cb45d563eb None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:39.857716 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:39.868483 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47964079-d462-49c1-8053-9fe1db625e6f None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:39.917571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1eece9-68c6-4742-8b80-7cf7e8625f88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:39.918591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1eece9-68c6-4742-8b80-7cf7e8625f88 None admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:39.919070 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1eece9-68c6-4742-8b80-7cf7e8625f88 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:39.919531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1eece9-68c6-4742-8b80-7cf7e8625f88 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:39.921159 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1eece9-68c6-4742-8b80-7cf7e8625f88 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:39.923041 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1eece9-68c6-4742-8b80-7cf7e8625f88 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:39.923524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3e369f-475b-4509-9aa9-48cb45d563eb None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:39.941366 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:39.981164 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a1837c-8967-495e-a2a8-cb3fe2a2718b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:39.981811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a1837c-8967-495e-a2a8-cb3fe2a2718b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:39.982095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a1837c-8967-495e-a2a8-cb3fe2a2718b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:39.982371 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a1837c-8967-495e-a2a8-cb3fe2a2718b None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:39.983642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a1837c-8967-495e-a2a8-cb3fe2a2718b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:41:39.983975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a1837c-8967-495e-a2a8-cb3fe2a2718b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:39.985584 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a1837c-8967-495e-a2a8-cb3fe2a2718b None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:39.996100 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 17/34] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:41:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:39.997481 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528ed662-31d8-47d0-81c6-04d67e2dce6b None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:39.997774 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528ed662-31d8-47d0-81c6-04d67e2dce6b None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:39.998055 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528ed662-31d8-47d0-81c6-04d67e2dce6b None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:40.000265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 18/35] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:41:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:41:40.001213 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eaccd46-ecda-4291-a1ae-f232b35d7fe2 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:40.001497 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eaccd46-ecda-4291-a1ae-f232b35d7fe2 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:40.001776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eaccd46-ecda-4291-a1ae-f232b35d7fe2 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:40.003170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 19/36] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:41:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:41:40.003926 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eaccd46-ecda-4291-a1ae-f232b35d7fe2 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:40.061995 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 18/37] 10.222.0.42 () {60 vars in 1131 bytes} [Sun Sep 18 00:41:39 2022] POST /identity/v3/roles => generated 219 bytes in 390 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:41:40.063049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1eece9-68c6-4742-8b80-7cf7e8625f88 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:40.131456 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eaccd46-ecda-4291-a1ae-f232b35d7fe2 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:40.171931 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d5ee66-bc24-41ea-8f9a-2f352b4ef229 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:40.172561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d5ee66-bc24-41ea-8f9a-2f352b4ef229 None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:40.172875 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d5ee66-bc24-41ea-8f9a-2f352b4ef229 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:40.173121 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d5ee66-bc24-41ea-8f9a-2f352b4ef229 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:40.173827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d5ee66-bc24-41ea-8f9a-2f352b4ef229 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:40.175590 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d5ee66-bc24-41ea-8f9a-2f352b4ef229 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:40.198517 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 20/38] 10.222.0.42 () {60 vars in 1131 bytes} [Sun Sep 18 00:41:40 2022] POST /identity/v3/roles => generated 225 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:41:40.198517 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf11c615-85a7-4249-9102-da6d834627e8 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:40.198517 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf11c615-85a7-4249-9102-da6d834627e8 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:40.198517 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf11c615-85a7-4249-9102-da6d834627e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:40.224566 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-bf11c615-85a7-4249-9102-da6d834627e8 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:41:40.227227 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-bf11c615-85a7-4249-9102-da6d834627e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:41:40.244173 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1eece9-68c6-4742-8b80-7cf7e8625f88 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:40.265887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:40.296080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d7765a-df55-4d8a-a84f-75226312b78b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:40.298781 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d7765a-df55-4d8a-a84f-75226312b78b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:40.299463 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d7765a-df55-4d8a-a84f-75226312b78b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:40.300097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d7765a-df55-4d8a-a84f-75226312b78b None admin] PATH_INFO: `/v3/users/6f5558a706da458791737ed054e51874` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:40.302240 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d7765a-df55-4d8a-a84f-75226312b78b None admin] RBAC: Authorizing `identity:get_user(user_id=6f5558a706da458791737ed054e51874)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:40.304836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d7765a-df55-4d8a-a84f-75226312b78b None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:40.307940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 19/39] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:41:40 2022] GET /identity/v3/users/6f5558a706da458791737ed054e51874 => generated 247 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:40.309398 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b510389-c981-4969-b574-a1671ac7d66b None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:40.309754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b510389-c981-4969-b574-a1671ac7d66b None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:40.310098 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b510389-c981-4969-b574-a1671ac7d66b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:40.332776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 21/40] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:41:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:41:40.333929 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf11c615-85a7-4249-9102-da6d834627e8 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:40.334146 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-0b510389-c981-4969-b574-a1671ac7d66b None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:41:40.335935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-0b510389-c981-4969-b574-a1671ac7d66b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:41:40.361969 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:40.440564 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 20/41] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:41:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:41:40.480975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf11c615-85a7-4249-9102-da6d834627e8 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:40.618997 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b510389-c981-4969-b574-a1671ac7d66b None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:40.630237 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a07ac3-7948-4dfe-8ddb-281300bb148a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:40.631233 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a07ac3-7948-4dfe-8ddb-281300bb148a None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:40.631521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a07ac3-7948-4dfe-8ddb-281300bb148a None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:40.632086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a07ac3-7948-4dfe-8ddb-281300bb148a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:40.633357 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a07ac3-7948-4dfe-8ddb-281300bb148a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:40.635499 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a07ac3-7948-4dfe-8ddb-281300bb148a None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:40.637267 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 22/42] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:41:40 2022] GET /identity/v3/domains/default => generated 199 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:40.643465 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a07ac3-7948-4dfe-8ddb-281300bb148a None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:40.674176 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:40.733339 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b510389-c981-4969-b574-a1671ac7d66b None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:40.763921 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a07ac3-7948-4dfe-8ddb-281300bb148a None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:40.769888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845b2654-3c68-4c9e-ba7f-c635339815eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:40.770534 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845b2654-3c68-4c9e-ba7f-c635339815eb None admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:40.770815 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845b2654-3c68-4c9e-ba7f-c635339815eb None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:40.771092 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845b2654-3c68-4c9e-ba7f-c635339815eb None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:40.771770 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845b2654-3c68-4c9e-ba7f-c635339815eb None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:40.773164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845b2654-3c68-4c9e-ba7f-c635339815eb None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:40.789487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 21/43] 10.222.0.42 () {60 vars in 1131 bytes} [Sun Sep 18 00:41:40 2022] POST /identity/v3/roles => generated 223 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:41:40.790235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845b2654-3c68-4c9e-ba7f-c635339815eb None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:40.810451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a51fa7-5663-4bb9-9ccf-2e01fcd53bcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:40.811499 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a51fa7-5663-4bb9-9ccf-2e01fcd53bcd None admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:40.811922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a51fa7-5663-4bb9-9ccf-2e01fcd53bcd None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:40.812348 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a51fa7-5663-4bb9-9ccf-2e01fcd53bcd None admin] PATH_INFO: `/v3/domains/default/users/6f5558a706da458791737ed054e51874/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:40.815077 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a51fa7-5663-4bb9-9ccf-2e01fcd53bcd None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=6f5558a706da458791737ed054e51874, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:40.817200 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a51fa7-5663-4bb9-9ccf-2e01fcd53bcd None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:40.823057 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:40.841916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 23/44] 10.222.0.42 () {58 vars in 1361 bytes} [Sun Sep 18 00:41:40 2022] PUT /identity/v3/domains/default/users/6f5558a706da458791737ed054e51874/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:41:40.964759 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845b2654-3c68-4c9e-ba7f-c635339815eb None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:41.016132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5bf408-4c05-4ce3-8823-af82dd102bed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:41.016859 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5bf408-4c05-4ce3-8823-af82dd102bed None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:41.017143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5bf408-4c05-4ce3-8823-af82dd102bed None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:41.017416 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5bf408-4c05-4ce3-8823-af82dd102bed None admin] PATH_INFO: `/v3/domains/default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:41.018783 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5bf408-4c05-4ce3-8823-af82dd102bed None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:41.020277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5bf408-4c05-4ce3-8823-af82dd102bed None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:41.021965 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 22/45] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:41:40 2022] GET /identity/v3/domains/default => generated 199 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:41.028440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a51fa7-5663-4bb9-9ccf-2e01fcd53bcd None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:41.126649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:41.139867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a51fa7-5663-4bb9-9ccf-2e01fcd53bcd None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:41.172546 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d256fef9-9945-40bc-8473-df231603bab7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:41.173155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d256fef9-9945-40bc-8473-df231603bab7 None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:41.173435 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d256fef9-9945-40bc-8473-df231603bab7 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:41.173710 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d256fef9-9945-40bc-8473-df231603bab7 None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:41.174653 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d256fef9-9945-40bc-8473-df231603bab7 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:41.176104 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d256fef9-9945-40bc-8473-df231603bab7 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:41.204342 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 24/46] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 18 00:41:41 2022] POST /identity/v3/projects => generated 312 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:41:43.038263 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9acff8-5401-4132-9ef8-d9a04cdc5501 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:43.038850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9acff8-5401-4132-9ef8-d9a04cdc5501 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:43.038850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9acff8-5401-4132-9ef8-d9a04cdc5501 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:43.040378 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 23/47] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:41:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:41:43.045745 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a394c6dd-7a3b-4be3-a9d5-23ff3aa9cf1d None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:43.046193 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a394c6dd-7a3b-4be3-a9d5-23ff3aa9cf1d None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:43.046307 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a394c6dd-7a3b-4be3-a9d5-23ff3aa9cf1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:43.066984 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-a394c6dd-7a3b-4be3-a9d5-23ff3aa9cf1d None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:41:43.069033 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a394c6dd-7a3b-4be3-a9d5-23ff3aa9cf1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:41:43.143235 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 25/48] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:41:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:41:43.412380 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9acff8-5401-4132-9ef8-d9a04cdc5501 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:43.418261 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980c7d94-9bfa-4def-afc5-a51e62ce420e None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:43.418585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980c7d94-9bfa-4def-afc5-a51e62ce420e None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:43.418869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980c7d94-9bfa-4def-afc5-a51e62ce420e None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:43.420305 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 26/49] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:41:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:41:43.426391 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c8644e-589b-4a70-83e3-8520c2b83b61 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:43.426705 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c8644e-589b-4a70-83e3-8520c2b83b61 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:43.426984 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c8644e-589b-4a70-83e3-8520c2b83b61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:43.448359 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-05c8644e-589b-4a70-83e3-8520c2b83b61 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:41:43.449606 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-05c8644e-589b-4a70-83e3-8520c2b83b61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:41:43.525844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:43.526801 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:43.533402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 27/50] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:41:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:41:43.549075 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9acff8-5401-4132-9ef8-d9a04cdc5501 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:43.589949 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdc9164-2725-42d4-8812-c5ae750708bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:43.590629 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdc9164-2725-42d4-8812-c5ae750708bf None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:43.590915 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdc9164-2725-42d4-8812-c5ae750708bf None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:43.591192 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdc9164-2725-42d4-8812-c5ae750708bf None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:43.602552 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bdc9164-2725-42d4-8812-c5ae750708bf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:41:43.607511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-1bdc9164-2725-42d4-8812-c5ae750708bf None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:41:43.607511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:41:43.607511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:41:43.607511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:41:43.607511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:41:43.607511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:41:43.607511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:41:43.607511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:41:43.607511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:41:43.607511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:41:43.607511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:41:43.607511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:41:43.607511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:41:43.607511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:41:43.607511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:41:43.607511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:41:43.609046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:41:43.609046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:41:43.609046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:41:43.609046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:41:43.609046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:41:43.609046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:41:43.609046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:41:43.609046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:41:43.609046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:41:43.609046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:41:43.609046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:41:43.609046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:41:43.609046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:41:43.609046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:41:43.609046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:41:43.609046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:41:43.609803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:41:43.609803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:41:43.609803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:41:43.609803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:41:43.609803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:41:43.609803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:41:43.609803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:41:43.609803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:41:43.609803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:41:43.609803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:41:43.609803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:41:43.609803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:41:43.609803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 24/51] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:41:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:41:43.615270 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c8644e-589b-4a70-83e3-8520c2b83b61 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:43.720553 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c8644e-589b-4a70-83e3-8520c2b83b61 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:43.753903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19fc7cf-998b-44c8-81cb-87cd6d59cf1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:43.754543 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19fc7cf-998b-44c8-81cb-87cd6d59cf1f None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:43.754822 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19fc7cf-998b-44c8-81cb-87cd6d59cf1f None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:43.755093 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19fc7cf-998b-44c8-81cb-87cd6d59cf1f None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:43.755758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19fc7cf-998b-44c8-81cb-87cd6d59cf1f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:41:43.756059 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19fc7cf-998b-44c8-81cb-87cd6d59cf1f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:43.757438 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19fc7cf-998b-44c8-81cb-87cd6d59cf1f None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:43.765465 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 28/52] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:41:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:43.771111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdc9164-2725-42d4-8812-c5ae750708bf None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:43.804013 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19fc7cf-998b-44c8-81cb-87cd6d59cf1f None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:43.873894 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdc9164-2725-42d4-8812-c5ae750708bf None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:43.876475 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:43.908166 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e030da16-8fa4-4b80-b2cc-332f35d2ecf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:43.908804 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e030da16-8fa4-4b80-b2cc-332f35d2ecf6 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:43.909082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e030da16-8fa4-4b80-b2cc-332f35d2ecf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:43.909358 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e030da16-8fa4-4b80-b2cc-332f35d2ecf6 None admin] PATH_INFO: `/v3/users/6f5558a706da458791737ed054e51874` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:43.910517 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e030da16-8fa4-4b80-b2cc-332f35d2ecf6 None admin] RBAC: Authorizing `identity:get_user(user_id=6f5558a706da458791737ed054e51874)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:43.912086 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e030da16-8fa4-4b80-b2cc-332f35d2ecf6 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:43.913542 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 25/53] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:41:43 2022] GET /identity/v3/users/6f5558a706da458791737ed054e51874 => generated 247 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:43.915690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19fc7cf-998b-44c8-81cb-87cd6d59cf1f None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:43.919358 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e030da16-8fa4-4b80-b2cc-332f35d2ecf6 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:43.933259 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:43.950658 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d01f8c-cf40-48bc-953f-d59ffde5c905 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:43.951296 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d01f8c-cf40-48bc-953f-d59ffde5c905 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:43.951611 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d01f8c-cf40-48bc-953f-d59ffde5c905 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:43.951892 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d01f8c-cf40-48bc-953f-d59ffde5c905 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:43.953019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d01f8c-cf40-48bc-953f-d59ffde5c905 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:43.954531 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d01f8c-cf40-48bc-953f-d59ffde5c905 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:43.956108 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 29/54] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:41:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:41:43.961622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d01f8c-cf40-48bc-953f-d59ffde5c905 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:44.019227 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e030da16-8fa4-4b80-b2cc-332f35d2ecf6 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:44.054105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a50656-0920-4df4-b9a9-b29025afdf22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:44.054796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a50656-0920-4df4-b9a9-b29025afdf22 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:44.055044 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a50656-0920-4df4-b9a9-b29025afdf22 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:44.055321 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a50656-0920-4df4-b9a9-b29025afdf22 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:44.056467 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a50656-0920-4df4-b9a9-b29025afdf22 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:44.057938 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a50656-0920-4df4-b9a9-b29025afdf22 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:44.059560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 26/55] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:41:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:44.061717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d01f8c-cf40-48bc-953f-d59ffde5c905 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:44.065413 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a50656-0920-4df4-b9a9-b29025afdf22 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:44.089896 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:44.097189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966c05a6-3d71-4aee-be03-0dcf5ab66f66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:44.097810 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966c05a6-3d71-4aee-be03-0dcf5ab66f66 None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:44.098094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966c05a6-3d71-4aee-be03-0dcf5ab66f66 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:44.098371 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966c05a6-3d71-4aee-be03-0dcf5ab66f66 None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:44.099063 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966c05a6-3d71-4aee-be03-0dcf5ab66f66 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:44.100520 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966c05a6-3d71-4aee-be03-0dcf5ab66f66 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:44.128918 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 30/56] 10.222.0.42 () {60 vars in 1140 bytes} [Sun Sep 18 00:41:43 2022] POST /identity/v3/projects => generated 298 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:41:44.151137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:44.179282 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a50656-0920-4df4-b9a9-b29025afdf22 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:44.216473 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7834f3de-8599-44ea-9e82-44019b75b9cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:44.217648 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7834f3de-8599-44ea-9e82-44019b75b9cd None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:44.217942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7834f3de-8599-44ea-9e82-44019b75b9cd None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:44.218217 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7834f3de-8599-44ea-9e82-44019b75b9cd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:44.218994 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7834f3de-8599-44ea-9e82-44019b75b9cd None admin] RBAC: Adding query filter params (role.id=1e04c9fea65244d28e1e59fc4c64f32f, scope.domain.id=default, user.id=6f5558a706da458791737ed054e51874) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:41:44.219307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7834f3de-8599-44ea-9e82-44019b75b9cd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:44.220802 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7834f3de-8599-44ea-9e82-44019b75b9cd None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:44.230685 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 27/57] 10.222.0.42 () {56 vars in 1322 bytes} [Sun Sep 18 00:41:44 2022] GET /identity/v3/role_assignments?user.id=6f5558a706da458791737ed054e51874&scope.domain.id=default&role.id=1e04c9fea65244d28e1e59fc4c64f32f => generated 530 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:41:46.320451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba007861-278c-448b-a104-95320edad99a None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:46.320971 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba007861-278c-448b-a104-95320edad99a None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:46.320971 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba007861-278c-448b-a104-95320edad99a None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:46.322345 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 31/58] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:41:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:41:46.327694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d921582-10f7-4778-b009-adc166533048 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:46.331371 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d921582-10f7-4778-b009-adc166533048 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:46.331689 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d921582-10f7-4778-b009-adc166533048 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:46.453471 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-1d921582-10f7-4778-b009-adc166533048 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:41:46.454719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-1d921582-10f7-4778-b009-adc166533048 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:41:46.521061 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 28/59] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:41:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:41:46.788401 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba007861-278c-448b-a104-95320edad99a None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:46.894786 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba007861-278c-448b-a104-95320edad99a None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:46.934141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a064d14-07f5-43cd-a25d-753bd488a0d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:46.935123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a064d14-07f5-43cd-a25d-753bd488a0d3 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:46.935549 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a064d14-07f5-43cd-a25d-753bd488a0d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:46.936000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a064d14-07f5-43cd-a25d-753bd488a0d3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:46.937736 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a064d14-07f5-43cd-a25d-753bd488a0d3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:46.940115 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a064d14-07f5-43cd-a25d-753bd488a0d3 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:46.942698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 32/60] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:41:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:46.948637 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d921582-10f7-4778-b009-adc166533048 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:47.025358 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:47.059018 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d921582-10f7-4778-b009-adc166533048 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:47.095114 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f39ab85-a19c-4c1a-b5ca-3efe7af62ffe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:47.095742 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f39ab85-a19c-4c1a-b5ca-3efe7af62ffe None admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:47.096023 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f39ab85-a19c-4c1a-b5ca-3efe7af62ffe None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:47.096297 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f39ab85-a19c-4c1a-b5ca-3efe7af62ffe None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:47.096978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f39ab85-a19c-4c1a-b5ca-3efe7af62ffe None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:47.098434 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f39ab85-a19c-4c1a-b5ca-3efe7af62ffe None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:47.130035 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 29/61] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 18 00:41:46 2022] POST /identity/v3/users => generated 275 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:41:51.408946 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d898b2-d983-4a85-b267-e2feeccafb42 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:51.409497 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d898b2-d983-4a85-b267-e2feeccafb42 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:51.409911 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d898b2-d983-4a85-b267-e2feeccafb42 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:51.412919 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 33/62] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:41:51 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:41:51.420520 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe8f4f0-6ae9-4663-99db-a896a242b564 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:51.421030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe8f4f0-6ae9-4663-99db-a896a242b564 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:51.421116 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe8f4f0-6ae9-4663-99db-a896a242b564 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:51.448191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-cfe8f4f0-6ae9-4663-99db-a896a242b564 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:41:51.449415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-cfe8f4f0-6ae9-4663-99db-a896a242b564 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:41:51.469392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa19119-b6bf-4242-8d29-bbda6e346cdd None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:51.469878 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa19119-b6bf-4242-8d29-bbda6e346cdd None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:51.470370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa19119-b6bf-4242-8d29-bbda6e346cdd None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:51.475683 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 34/63] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:41:51 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:41:51.484649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487d4258-4fac-4550-acd2-e3677f3bdfdf None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:51.485367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487d4258-4fac-4550-acd2-e3677f3bdfdf None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:51.486066 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487d4258-4fac-4550-acd2-e3677f3bdfdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:51.530349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-487d4258-4fac-4550-acd2-e3677f3bdfdf None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:41:51.534084 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-487d4258-4fac-4550-acd2-e3677f3bdfdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:41:51.578924 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 30/64] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:41:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:41:51.583327 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51320cf4-8743-45b3-9d16-8fb65b0804eb None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:51.584452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51320cf4-8743-45b3-9d16-8fb65b0804eb None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:51.585246 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51320cf4-8743-45b3-9d16-8fb65b0804eb None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:51.587212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 31/65] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:41:51 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:41:51.596869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f925bac2-8970-4eba-a6b5-4ca3be898ed6 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:51.597285 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f925bac2-8970-4eba-a6b5-4ca3be898ed6 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:51.599594 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f925bac2-8970-4eba-a6b5-4ca3be898ed6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:51.646314 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:51.649033 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-f925bac2-8970-4eba-a6b5-4ca3be898ed6 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:41:51.651169 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-f925bac2-8970-4eba-a6b5-4ca3be898ed6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:41:51.697224 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 35/66] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:41:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:41:51.735587 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:51.751374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 32/67] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:41:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:41:51.933018 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487d4258-4fac-4550-acd2-e3677f3bdfdf None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:52.006059 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f925bac2-8970-4eba-a6b5-4ca3be898ed6 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:52.105581 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487d4258-4fac-4550-acd2-e3677f3bdfdf None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:52.122050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f925bac2-8970-4eba-a6b5-4ca3be898ed6 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:52.151472 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7eb8c1-efa0-4215-b19e-c68d241dc08e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:52.152425 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7eb8c1-efa0-4215-b19e-c68d241dc08e None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:52.152533 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7eb8c1-efa0-4215-b19e-c68d241dc08e None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:52.152783 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7eb8c1-efa0-4215-b19e-c68d241dc08e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:52.164952 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c539ed5-72d3-4722-a47d-f80b1f96530d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:52.165786 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7eb8c1-efa0-4215-b19e-c68d241dc08e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:41:52.166214 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c539ed5-72d3-4722-a47d-f80b1f96530d None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:52.166734 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c539ed5-72d3-4722-a47d-f80b1f96530d None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:52.167211 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c539ed5-72d3-4722-a47d-f80b1f96530d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:52.171259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-1e7eb8c1-efa0-4215-b19e-c68d241dc08e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:41:52.171259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:41:52.171259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:41:52.171259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:41:52.171259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:41:52.171259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:41:52.171259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:41:52.171259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:41:52.171259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:41:52.171259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:41:52.171259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:41:52.171259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:41:52.171259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:41:52.171259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:41:52.171259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:41:52.171259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:41:52.172140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:41:52.172140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:41:52.172140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:41:52.172140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:41:52.172140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:41:52.172140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:41:52.172140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:41:52.172140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:41:52.172140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:41:52.172140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:41:52.172140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:41:52.172140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:41:52.172140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:41:52.172140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:41:52.172140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:41:52.172140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:41:52.173395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:41:52.173395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:41:52.173395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:41:52.173395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:41:52.173395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:41:52.173395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:41:52.173395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:41:52.173395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:41:52.173395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:41:52.173395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:41:52.173395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:41:52.173395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:41:52.173395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 36/68] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:41:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:41:52.173395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7eb8c1-efa0-4215-b19e-c68d241dc08e None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:52.184798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c539ed5-72d3-4722-a47d-f80b1f96530d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:41:52.192308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-4c539ed5-72d3-4722-a47d-f80b1f96530d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:41:52.192308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:41:52.192308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:41:52.192308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:41:52.192308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:41:52.192308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:41:52.192308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:41:52.192308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:41:52.192308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:41:52.192308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:41:52.192308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:41:52.192308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:41:52.192308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:41:52.192308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:41:52.192308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:41:52.192308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:41:52.193267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:41:52.193267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:41:52.193267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:41:52.193267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:41:52.193267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:41:52.193267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:41:52.193267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:41:52.193267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:41:52.193267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:41:52.193267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:41:52.193267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:41:52.193267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:41:52.193267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:41:52.193267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:41:52.193267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:41:52.193267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:41:52.194028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:41:52.194028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:41:52.194028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:41:52.194028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:41:52.194028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:41:52.194028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:41:52.194028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:41:52.194028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:41:52.194028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:41:52.194028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:41:52.194028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:41:52.194028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:41:52.194028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 33/69] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:41:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:41:52.198820 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c539ed5-72d3-4722-a47d-f80b1f96530d None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:52.251723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:52.280507 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:52.344025 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c539ed5-72d3-4722-a47d-f80b1f96530d None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:52.346450 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7eb8c1-efa0-4215-b19e-c68d241dc08e None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:52.387682 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be4a545-b7a0-498e-90f6-5dc3a75c03ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:52.388325 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be4a545-b7a0-498e-90f6-5dc3a75c03ed None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:52.388709 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be4a545-b7a0-498e-90f6-5dc3a75c03ed None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:52.388990 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be4a545-b7a0-498e-90f6-5dc3a75c03ed None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:52.400355 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be4a545-b7a0-498e-90f6-5dc3a75c03ed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:41:52.406308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-5be4a545-b7a0-498e-90f6-5dc3a75c03ed None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:41:52.406308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:41:52.406308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:41:52.406308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:41:52.406308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:41:52.406308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:41:52.406308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:41:52.406308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:41:52.406308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:41:52.406308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:41:52.406308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:41:52.406308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:41:52.406308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:41:52.406308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:41:52.406308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:41:52.406308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:41:52.407262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:41:52.407262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:41:52.407262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:41:52.407262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:41:52.407262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:41:52.407262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:41:52.407262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:41:52.407262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:41:52.407262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:41:52.407262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:41:52.407262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:41:52.407262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:41:52.407262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:41:52.407262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:41:52.407262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:41:52.407262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:41:52.408107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:41:52.408107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:41:52.408107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:41:52.408107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:41:52.408107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:41:52.408107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:41:52.408107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:41:52.408107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:41:52.408107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:41:52.408107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:41:52.408107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:41:52.408107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:41:52.408107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 37/70] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 18 00:41:52 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:41:52.408728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be4a545-b7a0-498e-90f6-5dc3a75c03ed None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:52.410498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd87e39-c9a6-4f45-85d1-f3c4a4724f71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:52.411505 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd87e39-c9a6-4f45-85d1-f3c4a4724f71 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:52.411994 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd87e39-c9a6-4f45-85d1-f3c4a4724f71 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:52.412471 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd87e39-c9a6-4f45-85d1-f3c4a4724f71 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:52.413663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd87e39-c9a6-4f45-85d1-f3c4a4724f71 None admin] RBAC: Adding query filter params (name=member) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:41:52.414207 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd87e39-c9a6-4f45-85d1-f3c4a4724f71 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:52.417292 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd87e39-c9a6-4f45-85d1-f3c4a4724f71 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:52.441479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 34/71] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:41:52 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:52.441479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478c164d-c7d4-4465-ac65-ca909b8ef562 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:52.441479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478c164d-c7d4-4465-ac65-ca909b8ef562 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:52.441479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478c164d-c7d4-4465-ac65-ca909b8ef562 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:52.441479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 35/72] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:41:52 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:41:52.449969 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094ef2e8-81cf-4c73-b428-5289cc34cc1b None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:52.449969 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094ef2e8-81cf-4c73-b428-5289cc34cc1b None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:52.450303 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094ef2e8-81cf-4c73-b428-5289cc34cc1b None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:52.453119 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 36/73] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:41:52 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:41:52.454246 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-094ef2e8-81cf-4c73-b428-5289cc34cc1b None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:52.541147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:52.545705 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be4a545-b7a0-498e-90f6-5dc3a75c03ed None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:52.587497 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2846cc17-d788-4b1a-b17b-554ce37b9b3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:52.588346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2846cc17-d788-4b1a-b17b-554ce37b9b3e None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:52.588670 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2846cc17-d788-4b1a-b17b-554ce37b9b3e None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:52.589119 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2846cc17-d788-4b1a-b17b-554ce37b9b3e None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:52.589863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2846cc17-d788-4b1a-b17b-554ce37b9b3e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:41:52.590246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2846cc17-d788-4b1a-b17b-554ce37b9b3e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:52.591942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-094ef2e8-81cf-4c73-b428-5289cc34cc1b None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:52.592105 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2846cc17-d788-4b1a-b17b-554ce37b9b3e None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:52.602174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 38/74] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:41:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:52.603396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2846cc17-d788-4b1a-b17b-554ce37b9b3e None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:52.606581 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:52.645170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9d25e4-5bfb-4dfe-bd62-9c2069c8b0fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:52.646249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9d25e4-5bfb-4dfe-bd62-9c2069c8b0fe None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:52.646806 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9d25e4-5bfb-4dfe-bd62-9c2069c8b0fe None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:52.647301 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9d25e4-5bfb-4dfe-bd62-9c2069c8b0fe None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:52.648531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9d25e4-5bfb-4dfe-bd62-9c2069c8b0fe None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:41:52.648954 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9d25e4-5bfb-4dfe-bd62-9c2069c8b0fe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:52.651586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9d25e4-5bfb-4dfe-bd62-9c2069c8b0fe None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:52.667156 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 37/75] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:41:52 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:52.669252 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffefd0d-2f43-4a36-a80f-367b054c1189 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:52.669757 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffefd0d-2f43-4a36-a80f-367b054c1189 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:52.670786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffefd0d-2f43-4a36-a80f-367b054c1189 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:52.691771 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-0ffefd0d-2f43-4a36-a80f-367b054c1189 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:41:52.693004 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-0ffefd0d-2f43-4a36-a80f-367b054c1189 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:41:52.735760 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2846cc17-d788-4b1a-b17b-554ce37b9b3e None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:52.775611 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e17d8d-9ca8-4646-a618-4207ca0c06a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:52.776279 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e17d8d-9ca8-4646-a618-4207ca0c06a8 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:52.776559 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e17d8d-9ca8-4646-a618-4207ca0c06a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:52.776836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e17d8d-9ca8-4646-a618-4207ca0c06a8 None admin] PATH_INFO: `/v3/users/796c8ea379cf4e7c9570bc2fdba53211` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:52.780062 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 38/76] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:41:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:41:52.788021 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda35256-a41e-408d-8595-05b08bbbaf2e None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:52.788308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda35256-a41e-408d-8595-05b08bbbaf2e None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:52.788587 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda35256-a41e-408d-8595-05b08bbbaf2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:52.800706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e17d8d-9ca8-4646-a618-4207ca0c06a8 None admin] RBAC: Authorizing `identity:get_user(user_id=796c8ea379cf4e7c9570bc2fdba53211)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:52.803372 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e17d8d-9ca8-4646-a618-4207ca0c06a8 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:52.806346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 39/77] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:41:52 2022] GET /identity/v3/users/796c8ea379cf4e7c9570bc2fdba53211 => generated 275 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:52.807825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e17d8d-9ca8-4646-a618-4207ca0c06a8 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:52.821776 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-bda35256-a41e-408d-8595-05b08bbbaf2e None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:41:52.823420 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-bda35256-a41e-408d-8595-05b08bbbaf2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:41:52.846677 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:52.953215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 39/78] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:41:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:41:52.954985 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda35256-a41e-408d-8595-05b08bbbaf2e None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:52.969188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e17d8d-9ca8-4646-a618-4207ca0c06a8 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:53.011076 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:53.016888 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfc8168-91e2-4a32-a7b6-ee5ba16312c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:53.017749 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfc8168-91e2-4a32-a7b6-ee5ba16312c3 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:53.018149 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfc8168-91e2-4a32-a7b6-ee5ba16312c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:53.018564 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfc8168-91e2-4a32-a7b6-ee5ba16312c3 None admin] PATH_INFO: `/v3/users/6f5558a706da458791737ed054e51874` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:53.020290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfc8168-91e2-4a32-a7b6-ee5ba16312c3 None admin] RBAC: Authorizing `identity:get_user(user_id=6f5558a706da458791737ed054e51874)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:53.022917 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfc8168-91e2-4a32-a7b6-ee5ba16312c3 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:53.030492 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 40/79] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:41:52 2022] GET /identity/v3/users/6f5558a706da458791737ed054e51874 => generated 247 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:53.030492 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfc8168-91e2-4a32-a7b6-ee5ba16312c3 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:53.126440 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda35256-a41e-408d-8595-05b08bbbaf2e None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:53.152204 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:53.168469 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfc8168-91e2-4a32-a7b6-ee5ba16312c3 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:53.191520 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b796e80-759e-403c-ac92-a0abd8e88edc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:53.192708 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b796e80-759e-403c-ac92-a0abd8e88edc None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:53.193003 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b796e80-759e-403c-ac92-a0abd8e88edc None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:53.193284 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b796e80-759e-403c-ac92-a0abd8e88edc None admin] PATH_INFO: `/v3/users/796c8ea379cf4e7c9570bc2fdba53211` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:53.194655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b796e80-759e-403c-ac92-a0abd8e88edc None admin] RBAC: Authorizing `identity:get_user(user_id=796c8ea379cf4e7c9570bc2fdba53211)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:53.196193 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b796e80-759e-403c-ac92-a0abd8e88edc None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:53.203640 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 40/80] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:41:52 2022] GET /identity/v3/users/796c8ea379cf4e7c9570bc2fdba53211 => generated 275 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:53.203640 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b796e80-759e-403c-ac92-a0abd8e88edc None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:53.223737 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097d4dc3-0441-4fb1-b344-13fed57799b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:53.224796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097d4dc3-0441-4fb1-b344-13fed57799b1 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:53.225289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097d4dc3-0441-4fb1-b344-13fed57799b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:53.225762 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097d4dc3-0441-4fb1-b344-13fed57799b1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:53.324451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097d4dc3-0441-4fb1-b344-13fed57799b1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:53.327026 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097d4dc3-0441-4fb1-b344-13fed57799b1 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:53.328875 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:53.334527 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b796e80-759e-403c-ac92-a0abd8e88edc None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:53.377501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21469d87-4441-4b91-9b6f-8680aeea3eb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:53.378058 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21469d87-4441-4b91-9b6f-8680aeea3eb7 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:53.378352 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21469d87-4441-4b91-9b6f-8680aeea3eb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:53.378641 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21469d87-4441-4b91-9b6f-8680aeea3eb7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:53.434428 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:53.453471 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 41/81] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:41:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:41:53.454354 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097d4dc3-0441-4fb1-b344-13fed57799b1 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:53.490992 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21469d87-4441-4b91-9b6f-8680aeea3eb7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:53.493830 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21469d87-4441-4b91-9b6f-8680aeea3eb7 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:53.574099 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097d4dc3-0441-4fb1-b344-13fed57799b1 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:53.626120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97cb3ca-b3f7-4f44-8173-aa8d665329c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:53.627158 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97cb3ca-b3f7-4f44-8173-aa8d665329c3 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:53.627657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97cb3ca-b3f7-4f44-8173-aa8d665329c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:53.627958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97cb3ca-b3f7-4f44-8173-aa8d665329c3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:53.632656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:53.639140 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 41/82] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:41:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:41:53.641004 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21469d87-4441-4b91-9b6f-8680aeea3eb7 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:53.646345 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97cb3ca-b3f7-4f44-8173-aa8d665329c3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:41:53.655979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-c97cb3ca-b3f7-4f44-8173-aa8d665329c3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:41:53.655979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:41:53.655979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:41:53.655979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:41:53.655979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:41:53.655979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:41:53.655979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:41:53.655979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:41:53.655979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:41:53.655979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:41:53.655979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:41:53.655979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:41:53.655979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:41:53.655979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:41:53.655979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:41:53.655979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:41:53.657259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:41:53.657259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:41:53.657259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:41:53.657259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:41:53.657259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:41:53.657259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:41:53.657259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:41:53.657259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:41:53.657259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:41:53.657259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:41:53.657259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:41:53.657259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:41:53.657259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:41:53.657259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:41:53.657259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:41:53.657259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:41:53.658320 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:41:53.658320 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:41:53.658320 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:41:53.658320 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:41:53.658320 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:41:53.658320 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:41:53.658320 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:41:53.658320 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:41:53.658320 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:41:53.658320 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:41:53.658320 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:41:53.658320 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:41:53.658320 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 42/83] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:41:53 2022] GET /identity/v3/roles/member => generated 84 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:41:53.660562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97cb3ca-b3f7-4f44-8173-aa8d665329c3 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:53.731361 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:53.774070 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21469d87-4441-4b91-9b6f-8680aeea3eb7 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:53.814403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a4a838-9b5b-440f-be25-523724e4098b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:53.815064 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a4a838-9b5b-440f-be25-523724e4098b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:53.815350 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a4a838-9b5b-440f-be25-523724e4098b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:53.815631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a4a838-9b5b-440f-be25-523724e4098b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:53.818858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97cb3ca-b3f7-4f44-8173-aa8d665329c3 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:53.868071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266414a3-1785-405a-b3a3-81c6c436752a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:53.869125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266414a3-1785-405a-b3a3-81c6c436752a None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:53.869605 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266414a3-1785-405a-b3a3-81c6c436752a None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:53.870078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266414a3-1785-405a-b3a3-81c6c436752a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:53.872092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266414a3-1785-405a-b3a3-81c6c436752a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:53.875365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266414a3-1785-405a-b3a3-81c6c436752a None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:53.878146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 43/84] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:41:53 2022] GET /identity/v3/domains/default => generated 199 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:53.879514 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266414a3-1785-405a-b3a3-81c6c436752a None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:53.894215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:53.898392 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a4a838-9b5b-440f-be25-523724e4098b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:53.901306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a4a838-9b5b-440f-be25-523724e4098b None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:54.010498 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266414a3-1785-405a-b3a3-81c6c436752a None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:54.030572 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:54.038297 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 42/85] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:41:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:41:54.040132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a4a838-9b5b-440f-be25-523724e4098b None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:54.052410 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1daad2f1-6fc7-4b35-8a2c-96c49cba9a9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:54.053192 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1daad2f1-6fc7-4b35-8a2c-96c49cba9a9e None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:54.053588 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1daad2f1-6fc7-4b35-8a2c-96c49cba9a9e None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:54.054008 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1daad2f1-6fc7-4b35-8a2c-96c49cba9a9e None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:54.055350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1daad2f1-6fc7-4b35-8a2c-96c49cba9a9e None admin] RBAC: Authorizing `identity:get_project(project_id=4c46ff2118f948a085f639b37068aa57)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:54.057039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1daad2f1-6fc7-4b35-8a2c-96c49cba9a9e None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:54.058947 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 44/86] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:41:53 2022] GET /identity/v3/projects/4c46ff2118f948a085f639b37068aa57 => generated 298 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:54.060243 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1daad2f1-6fc7-4b35-8a2c-96c49cba9a9e None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:54.149329 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:54.152606 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a4a838-9b5b-440f-be25-523724e4098b None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:54.197373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1daad2f1-6fc7-4b35-8a2c-96c49cba9a9e None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:54.198004 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe0dc86-e964-4421-882c-1e60c9602595 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:54.199043 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe0dc86-e964-4421-882c-1e60c9602595 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:54.199519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe0dc86-e964-4421-882c-1e60c9602595 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:54.199916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe0dc86-e964-4421-882c-1e60c9602595 None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:54.202017 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe0dc86-e964-4421-882c-1e60c9602595 None admin] RBAC: Authorizing `identity:get_project(project_id=4c46ff2118f948a085f639b37068aa57)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:54.204586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe0dc86-e964-4421-882c-1e60c9602595 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:54.212936 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 43/87] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:41:54 2022] GET /identity/v3/projects/4c46ff2118f948a085f639b37068aa57 => generated 298 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:54.212936 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe0dc86-e964-4421-882c-1e60c9602595 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:54.243824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f435c7-afc3-4487-b6bf-a027adffcd3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:54.244605 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f435c7-afc3-4487-b6bf-a027adffcd3f None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:54.245068 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f435c7-afc3-4487-b6bf-a027adffcd3f None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:54.245455 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f435c7-afc3-4487-b6bf-a027adffcd3f None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:54.246422 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f435c7-afc3-4487-b6bf-a027adffcd3f None admin] RBAC: Adding query filter params (name=member) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:41:54.247048 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f435c7-afc3-4487-b6bf-a027adffcd3f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:54.248663 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f435c7-afc3-4487-b6bf-a027adffcd3f None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:54.258451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 45/88] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:41:54 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:54.259568 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f435c7-afc3-4487-b6bf-a027adffcd3f None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:54.320691 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:54.342095 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe0dc86-e964-4421-882c-1e60c9602595 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:54.391852 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a99d86c-05e5-4c02-a81e-3ca3c1463961 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:54.392638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a99d86c-05e5-4c02-a81e-3ca3c1463961 None admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:54.392981 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a99d86c-05e5-4c02-a81e-3ca3c1463961 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:54.393442 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a99d86c-05e5-4c02-a81e-3ca3c1463961 None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:54.394231 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a99d86c-05e5-4c02-a81e-3ca3c1463961 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:54.396050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a99d86c-05e5-4c02-a81e-3ca3c1463961 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:54.399908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f435c7-afc3-4487-b6bf-a027adffcd3f None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:54.423153 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 44/89] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 18 00:41:54 2022] POST /identity/v3/users => generated 289 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:41:54.424462 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a99d86c-05e5-4c02-a81e-3ca3c1463961 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:54.453986 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332faec2-79da-48cc-ac48-0c709d0c2672 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:54.455037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332faec2-79da-48cc-ac48-0c709d0c2672 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:54.455479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332faec2-79da-48cc-ac48-0c709d0c2672 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:54.455914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332faec2-79da-48cc-ac48-0c709d0c2672 None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:54.457844 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332faec2-79da-48cc-ac48-0c709d0c2672 None admin] RBAC: Authorizing `identity:get_project(project_id=4c46ff2118f948a085f639b37068aa57)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:54.460395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332faec2-79da-48cc-ac48-0c709d0c2672 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:54.469097 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 46/90] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:41:54 2022] GET /identity/v3/projects/4c46ff2118f948a085f639b37068aa57 => generated 298 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:54.469097 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332faec2-79da-48cc-ac48-0c709d0c2672 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:54.479192 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:54.636055 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a99d86c-05e5-4c02-a81e-3ca3c1463961 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:54.652009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332faec2-79da-48cc-ac48-0c709d0c2672 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:54.677389 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:54.696992 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b20ee97-1dd6-4f0f-b125-c9316cf2d58d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:54.698079 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b20ee97-1dd6-4f0f-b125-c9316cf2d58d None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:54.698254 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b20ee97-1dd6-4f0f-b125-c9316cf2d58d None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:54.698659 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b20ee97-1dd6-4f0f-b125-c9316cf2d58d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:54.702887 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b20ee97-1dd6-4f0f-b125-c9316cf2d58d None admin] RBAC: Adding query filter params (role.id=18f6cee478e54567b70d5c7557fe694e, scope.project.id=4c46ff2118f948a085f639b37068aa57, user.id=796c8ea379cf4e7c9570bc2fdba53211) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:41:54.702887 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b20ee97-1dd6-4f0f-b125-c9316cf2d58d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:54.702887 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b20ee97-1dd6-4f0f-b125-c9316cf2d58d None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:54.711066 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 45/91] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:41:54 2022] GET /identity/v3/role_assignments?user.id=796c8ea379cf4e7c9570bc2fdba53211&scope.project.id=4c46ff2118f948a085f639b37068aa57&role.id=18f6cee478e54567b70d5c7557fe694e => generated 259 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:54.711289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478f1a99-ce9f-41b2-8317-3121dde2c126 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:54.714512 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478f1a99-ce9f-41b2-8317-3121dde2c126 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:54.714683 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b20ee97-1dd6-4f0f-b125-c9316cf2d58d None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:54.714814 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478f1a99-ce9f-41b2-8317-3121dde2c126 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:54.714814 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478f1a99-ce9f-41b2-8317-3121dde2c126 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:54.714814 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478f1a99-ce9f-41b2-8317-3121dde2c126 None admin] RBAC: Adding query filter params (role.id=1e04c9fea65244d28e1e59fc4c64f32f, scope.project.id=4c46ff2118f948a085f639b37068aa57, user.id=6f5558a706da458791737ed054e51874) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:41:54.715176 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478f1a99-ce9f-41b2-8317-3121dde2c126 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:54.717748 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478f1a99-ce9f-41b2-8317-3121dde2c126 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:54.729578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 47/92] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:41:54 2022] GET /identity/v3/role_assignments?user.id=6f5558a706da458791737ed054e51874&scope.project.id=4c46ff2118f948a085f639b37068aa57&role.id=1e04c9fea65244d28e1e59fc4c64f32f => generated 259 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:54.731295 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478f1a99-ce9f-41b2-8317-3121dde2c126 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:54.891559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:54.908561 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b20ee97-1dd6-4f0f-b125-c9316cf2d58d None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:54.955273 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1620f488-310d-4d40-9a2e-f93b26fa52f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:54.955937 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1620f488-310d-4d40-9a2e-f93b26fa52f4 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:54.956244 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1620f488-310d-4d40-9a2e-f93b26fa52f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:54.956485 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1620f488-310d-4d40-9a2e-f93b26fa52f4 None admin] PATH_INFO: `/v3/users/796c8ea379cf4e7c9570bc2fdba53211` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:54.957840 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1620f488-310d-4d40-9a2e-f93b26fa52f4 None admin] RBAC: Authorizing `identity:get_user(user_id=796c8ea379cf4e7c9570bc2fdba53211)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:54.958871 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478f1a99-ce9f-41b2-8317-3121dde2c126 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:54.959372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1620f488-310d-4d40-9a2e-f93b26fa52f4 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:54.961127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 46/93] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:41:54 2022] GET /identity/v3/users/796c8ea379cf4e7c9570bc2fdba53211 => generated 275 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:54.967781 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1620f488-310d-4d40-9a2e-f93b26fa52f4 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:55.039337 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3efe204-bf3b-4d13-a44f-a79d607db154 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:55.040237 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3efe204-bf3b-4d13-a44f-a79d607db154 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:55.040563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3efe204-bf3b-4d13-a44f-a79d607db154 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:55.040921 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3efe204-bf3b-4d13-a44f-a79d607db154 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:55.041710 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3efe204-bf3b-4d13-a44f-a79d607db154 None admin] RBAC: Adding query filter params (role.id=cf934cf252ce42d5a0445d3e156348d1, scope.project.id=4c46ff2118f948a085f639b37068aa57, user.id=796c8ea379cf4e7c9570bc2fdba53211) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:41:55.042026 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3efe204-bf3b-4d13-a44f-a79d607db154 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:55.043587 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3efe204-bf3b-4d13-a44f-a79d607db154 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:55.056189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 48/94] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:41:54 2022] GET /identity/v3/role_assignments?user.id=796c8ea379cf4e7c9570bc2fdba53211&scope.project.id=4c46ff2118f948a085f639b37068aa57&role.id=cf934cf252ce42d5a0445d3e156348d1 => generated 259 bytes in 326 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:55.135750 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1620f488-310d-4d40-9a2e-f93b26fa52f4 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:55.183598 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1434fbae-9e18-4a9d-9c85-95bd98fa28fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:55.184563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1434fbae-9e18-4a9d-9c85-95bd98fa28fb None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:55.185015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1434fbae-9e18-4a9d-9c85-95bd98fa28fb None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:55.185979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1434fbae-9e18-4a9d-9c85-95bd98fa28fb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:55.224436 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:55.270266 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1434fbae-9e18-4a9d-9c85-95bd98fa28fb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:55.271959 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1434fbae-9e18-4a9d-9c85-95bd98fa28fb None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:55.386666 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 47/95] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:41:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 18 00:41:55.392668 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3efe204-bf3b-4d13-a44f-a79d607db154 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:55.476223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:55.569153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3efe204-bf3b-4d13-a44f-a79d607db154 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:55.627982 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0ee508-b9d7-4afe-a8e7-741726037e36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:55.629101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0ee508-b9d7-4afe-a8e7-741726037e36 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:55.629614 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0ee508-b9d7-4afe-a8e7-741726037e36 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:55.630117 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0ee508-b9d7-4afe-a8e7-741726037e36 None admin] PATH_INFO: `/v3/projects/c3a25b7f689e4e21b2f61ed886a2a2d2` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:55.636767 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0ee508-b9d7-4afe-a8e7-741726037e36 None admin] RBAC: Authorizing `identity:get_project(project_id=c3a25b7f689e4e21b2f61ed886a2a2d2)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:55.640087 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0ee508-b9d7-4afe-a8e7-741726037e36 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:55.643329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 49/96] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:41:55 2022] GET /identity/v3/projects/c3a25b7f689e4e21b2f61ed886a2a2d2 => generated 312 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:55.649632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1434fbae-9e18-4a9d-9c85-95bd98fa28fb None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:55.756404 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:55.811156 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1434fbae-9e18-4a9d-9c85-95bd98fa28fb None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:55.850248 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1bfeb4-72bc-4ab8-ae49-77fc8ef960bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:55.851303 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1bfeb4-72bc-4ab8-ae49-77fc8ef960bf None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:55.851787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1bfeb4-72bc-4ab8-ae49-77fc8ef960bf None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:55.852380 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1bfeb4-72bc-4ab8-ae49-77fc8ef960bf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:55.853357 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1bfeb4-72bc-4ab8-ae49-77fc8ef960bf None admin] RBAC: Adding query filter params (role.id=18f6cee478e54567b70d5c7557fe694e, scope.project.id=c3a25b7f689e4e21b2f61ed886a2a2d2, user.id=796c8ea379cf4e7c9570bc2fdba53211) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:41:55.853883 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1bfeb4-72bc-4ab8-ae49-77fc8ef960bf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:55.855742 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1bfeb4-72bc-4ab8-ae49-77fc8ef960bf None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:55.862418 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 48/97] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:41:55 2022] GET /identity/v3/role_assignments?user.id=796c8ea379cf4e7c9570bc2fdba53211&scope.project.id=c3a25b7f689e4e21b2f61ed886a2a2d2&role.id=18f6cee478e54567b70d5c7557fe694e => generated 259 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:57.934028 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120238e9-6f63-4c7c-a03d-3ec5d7767148 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:57.935676 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120238e9-6f63-4c7c-a03d-3ec5d7767148 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:57.936476 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120238e9-6f63-4c7c-a03d-3ec5d7767148 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:57.939385 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 50/98] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:41:57 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:41:57.948425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6ea6e6-21ec-4e58-bce8-aedb0df89d75 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:57.949871 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6ea6e6-21ec-4e58-bce8-aedb0df89d75 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:57.952857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6ea6e6-21ec-4e58-bce8-aedb0df89d75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:57.990877 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-2f6ea6e6-21ec-4e58-bce8-aedb0df89d75 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:41:57.993410 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-2f6ea6e6-21ec-4e58-bce8-aedb0df89d75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:41:58.163841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 49/99] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:41:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:41:58.839738 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ca4dca-9e44-4f20-9ab0-eca25f4fc066 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:58.840682 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ca4dca-9e44-4f20-9ab0-eca25f4fc066 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:58.841522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ca4dca-9e44-4f20-9ab0-eca25f4fc066 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:58.844413 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 51/100] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:41:58 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:41:58.852838 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e94f74-d817-49a1-bbd4-7937968a0960 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:58.853270 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e94f74-d817-49a1-bbd4-7937968a0960 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:58.853696 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e94f74-d817-49a1-bbd4-7937968a0960 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:58.887749 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-46e94f74-d817-49a1-bbd4-7937968a0960 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:41:58.889182 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-46e94f74-d817-49a1-bbd4-7937968a0960 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:41:58.934592 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16279c15-291e-4c32-b75e-bfcbef56a4c2 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:58.935371 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16279c15-291e-4c32-b75e-bfcbef56a4c2 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:58.935371 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16279c15-291e-4c32-b75e-bfcbef56a4c2 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:58.937239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 52/101] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:41:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:41:58.943001 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e8f903-529d-47d2-85e3-3a45488b5702 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:58.943379 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e8f903-529d-47d2-85e3-3a45488b5702 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:58.943600 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e8f903-529d-47d2-85e3-3a45488b5702 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:58.982131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-99e8f903-529d-47d2-85e3-3a45488b5702 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:41:58.984668 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-99e8f903-529d-47d2-85e3-3a45488b5702 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:41:58.991522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:59.030311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 50/102] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:41:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:41:59.122212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e94f74-d817-49a1-bbd4-7937968a0960 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:59.246240 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 53/103] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:41:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 302 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:41:59.346048 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e94f74-d817-49a1-bbd4-7937968a0960 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:59.403386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dbb8fa-f30c-46ab-bab9-d9bc0cf56d56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:59.403999 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dbb8fa-f30c-46ab-bab9-d9bc0cf56d56 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:59.404330 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dbb8fa-f30c-46ab-bab9-d9bc0cf56d56 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:59.404619 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dbb8fa-f30c-46ab-bab9-d9bc0cf56d56 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:59.405909 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85dbb8fa-f30c-46ab-bab9-d9bc0cf56d56 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:59.407468 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85dbb8fa-f30c-46ab-bab9-d9bc0cf56d56 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:59.409214 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 51/104] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:41:59 2022] GET /identity/v3/domains/default => generated 199 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:41:59.415933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e8f903-529d-47d2-85e3-3a45488b5702 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:59.527100 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:59.539558 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dbb8fa-f30c-46ab-bab9-d9bc0cf56d56 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:59.636323 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:59.689388 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e8f903-529d-47d2-85e3-3a45488b5702 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:59.743251 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d1cad3-e3ea-4a92-a6b7-2fdd1121e2f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:59.744168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d1cad3-e3ea-4a92-a6b7-2fdd1121e2f4 None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:59.744604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d1cad3-e3ea-4a92-a6b7-2fdd1121e2f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:59.745039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d1cad3-e3ea-4a92-a6b7-2fdd1121e2f4 None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:59.745875 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d1cad3-e3ea-4a92-a6b7-2fdd1121e2f4 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:41:59.747247 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dbb8fa-f30c-46ab-bab9-d9bc0cf56d56 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:59.748312 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d1cad3-e3ea-4a92-a6b7-2fdd1121e2f4 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:41:59.782665 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 54/105] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 18 00:41:59 2022] POST /identity/v3/projects => generated 302 bytes in 368 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:41:59.792180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817d1ed0-7b2c-43ab-8dc3-1da99e6bf6ce None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:59.793023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817d1ed0-7b2c-43ab-8dc3-1da99e6bf6ce None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:59.793658 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817d1ed0-7b2c-43ab-8dc3-1da99e6bf6ce None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:59.795786 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 55/106] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:41:59 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:41:59.796824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817d1ed0-7b2c-43ab-8dc3-1da99e6bf6ce None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:41:59.844647 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db815e91-421b-437c-ab22-56f096ad00ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:41:59.845865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db815e91-421b-437c-ab22-56f096ad00ff None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:59.846485 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db815e91-421b-437c-ab22-56f096ad00ff None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:59.847100 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db815e91-421b-437c-ab22-56f096ad00ff None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:59.891623 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db815e91-421b-437c-ab22-56f096ad00ff None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:41:59.909655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-db815e91-421b-437c-ab22-56f096ad00ff None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:41:59.909655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:41:59.909655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:41:59.909655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:41:59.909655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:41:59.909655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:41:59.909655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:41:59.909655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:41:59.909655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:41:59.909655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:41:59.909655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:41:59.909655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:41:59.909655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:41:59.909655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:41:59.909655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:41:59.909655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:41:59.910679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:41:59.910679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:41:59.910679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:41:59.910679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:41:59.910679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:41:59.910679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:41:59.910679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:41:59.910679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:41:59.910679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:41:59.910679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:41:59.910679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:41:59.910679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:41:59.910679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:41:59.910679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:41:59.910679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:41:59.910679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:41:59.911495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:41:59.911495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:41:59.911495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:41:59.911495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:41:59.911495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:41:59.911495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:41:59.911495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:41:59.911495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:41:59.911495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:41:59.911495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:41:59.911495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:41:59.911495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:41:59.912099 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 52/107] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:41:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 373 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:41:59.918344 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64385910-2cbe-4ff9-a7bf-956de3cf2ed5 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:41:59.918344 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64385910-2cbe-4ff9-a7bf-956de3cf2ed5 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:41:59.918344 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64385910-2cbe-4ff9-a7bf-956de3cf2ed5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:41:59.936562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:41:59.949981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817d1ed0-7b2c-43ab-8dc3-1da99e6bf6ce None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:41:59.984535 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-64385910-2cbe-4ff9-a7bf-956de3cf2ed5 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:41:59.986959 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-64385910-2cbe-4ff9-a7bf-956de3cf2ed5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:00.010638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f764fb1-2714-433c-9607-e1c1b5dd291b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:00.011626 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f764fb1-2714-433c-9607-e1c1b5dd291b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:00.012067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f764fb1-2714-433c-9607-e1c1b5dd291b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:00.012495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f764fb1-2714-433c-9607-e1c1b5dd291b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:00.027837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f764fb1-2714-433c-9607-e1c1b5dd291b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:00.047700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-4f764fb1-2714-433c-9607-e1c1b5dd291b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 18 00:42:00.047700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:00.047700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:00.047700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:00.047700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:00.047700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:00.047700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:00.047700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:00.047700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:00.047700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:00.047700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:00.047700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:00.047700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:00.047700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:00.047700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:00.047700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:00.048823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:00.048823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:00.048823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:00.048823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:00.048823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:00.048823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:00.048823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:00.048823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:00.048823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:00.048823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:00.048823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:00.048823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:00.048823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:00.048823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:00.048823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:00.048823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:00.050082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:00.050082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:00.050082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:00.050082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:00.050082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:00.050082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:00.050082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:00.050082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:00.050082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:00.050082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:00.050082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 18 00:42:00.050082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:42:00.057164 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 56/108] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:41:59 2022] GET /identity/v3/roles/reader => generated 84 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:00.058489 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f764fb1-2714-433c-9607-e1c1b5dd291b None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:00.161429 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 53/109] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:41:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 249 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:00.163106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64385910-2cbe-4ff9-a7bf-956de3cf2ed5 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:00.232676 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:00.235155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f764fb1-2714-433c-9607-e1c1b5dd291b None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:00.299688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77bc342-53bc-4e09-8103-94c89f441753 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:00.300342 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77bc342-53bc-4e09-8103-94c89f441753 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:00.300632 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77bc342-53bc-4e09-8103-94c89f441753 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:00.300914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77bc342-53bc-4e09-8103-94c89f441753 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:00.301623 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77bc342-53bc-4e09-8103-94c89f441753 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:00.301937 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77bc342-53bc-4e09-8103-94c89f441753 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:00.303462 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77bc342-53bc-4e09-8103-94c89f441753 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:00.313292 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 57/110] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:00.314643 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb79590f-4c4b-46ca-bd02-abd66ea3aa6c None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:00.314938 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb79590f-4c4b-46ca-bd02-abd66ea3aa6c None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:00.315375 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb79590f-4c4b-46ca-bd02-abd66ea3aa6c None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:00.317140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 58/111] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:00.318913 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b33eb3-ea57-4149-a9ff-2d3ee1af0925 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:00.319210 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b33eb3-ea57-4149-a9ff-2d3ee1af0925 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:00.319495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b33eb3-ea57-4149-a9ff-2d3ee1af0925 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:00.320893 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 59/112] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:00.323736 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b33eb3-ea57-4149-a9ff-2d3ee1af0925 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:00.336956 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64385910-2cbe-4ff9-a7bf-956de3cf2ed5 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:00.360820 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:00.395491 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0802dc9b-aa1b-4a7b-82c3-d35b8c76bbd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:00.397058 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0802dc9b-aa1b-4a7b-82c3-d35b8c76bbd0 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:00.397797 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0802dc9b-aa1b-4a7b-82c3-d35b8c76bbd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:00.398894 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0802dc9b-aa1b-4a7b-82c3-d35b8c76bbd0 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:00.401935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0802dc9b-aa1b-4a7b-82c3-d35b8c76bbd0 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:00.402787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0802dc9b-aa1b-4a7b-82c3-d35b8c76bbd0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:00.406207 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0802dc9b-aa1b-4a7b-82c3-d35b8c76bbd0 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:00.429556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 54/113] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:00 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:00.432078 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cea892-5882-4966-8a47-c7189a28254e None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:00.432700 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cea892-5882-4966-8a47-c7189a28254e None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:00.433324 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cea892-5882-4966-8a47-c7189a28254e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:00.472435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-11cea892-5882-4966-8a47-c7189a28254e None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:00.473682 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-11cea892-5882-4966-8a47-c7189a28254e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:00.523275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b33eb3-ea57-4149-a9ff-2d3ee1af0925 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:00.557379 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 55/114] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:00.558697 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83f332f-03eb-43b4-bf26-89b7049e41ca None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:00.558987 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83f332f-03eb-43b4-bf26-89b7049e41ca None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:00.559261 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83f332f-03eb-43b4-bf26-89b7049e41ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:00.579657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c593af6a-20d0-4b89-8ca7-7bac5a903879 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:00.581307 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c593af6a-20d0-4b89-8ca7-7bac5a903879 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:00.581778 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-d83f332f-03eb-43b4-bf26-89b7049e41ca None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:00.582277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c593af6a-20d0-4b89-8ca7-7bac5a903879 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:00.583025 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d83f332f-03eb-43b4-bf26-89b7049e41ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:00.583253 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c593af6a-20d0-4b89-8ca7-7bac5a903879 None admin] PATH_INFO: `/v3/users/6f5558a706da458791737ed054e51874` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:00.585754 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c593af6a-20d0-4b89-8ca7-7bac5a903879 None admin] RBAC: Authorizing `identity:get_user(user_id=6f5558a706da458791737ed054e51874)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:00.589093 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c593af6a-20d0-4b89-8ca7-7bac5a903879 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:00.593569 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 60/115] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:00 2022] GET /identity/v3/users/6f5558a706da458791737ed054e51874 => generated 247 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:00.595489 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c593af6a-20d0-4b89-8ca7-7bac5a903879 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:00.618371 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:00.667055 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 56/116] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:00.669843 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83f332f-03eb-43b4-bf26-89b7049e41ca None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:00.830442 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:00.897904 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c593af6a-20d0-4b89-8ca7-7bac5a903879 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:00.922195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83f332f-03eb-43b4-bf26-89b7049e41ca None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:00.965001 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7dd4f7c-42e0-470f-a66a-5b0e46200e77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:00.965986 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dd4f7c-42e0-470f-a66a-5b0e46200e77 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:00.966574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dd4f7c-42e0-470f-a66a-5b0e46200e77 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:00.967056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dd4f7c-42e0-470f-a66a-5b0e46200e77 None admin] PATH_INFO: `/v3/users/42fc7c5de9eb4a7cb41e4c636c8148e3` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:00.992090 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7dd4f7c-42e0-470f-a66a-5b0e46200e77 None admin] RBAC: Authorizing `identity:get_user(user_id=42fc7c5de9eb4a7cb41e4c636c8148e3)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:00.994967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7dd4f7c-42e0-470f-a66a-5b0e46200e77 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:00.998171 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 61/117] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:00 2022] GET /identity/v3/users/42fc7c5de9eb4a7cb41e4c636c8148e3 => generated 289 bytes in 404 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:00.999848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7dd4f7c-42e0-470f-a66a-5b0e46200e77 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:01.001389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb41144-7f0c-448a-bbee-6688b8fa2bc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:01.002392 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb41144-7f0c-448a-bbee-6688b8fa2bc5 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:01.002880 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb41144-7f0c-448a-bbee-6688b8fa2bc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:01.003326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb41144-7f0c-448a-bbee-6688b8fa2bc5 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:01.027604 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb41144-7f0c-448a-bbee-6688b8fa2bc5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:01.036342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-aeb41144-7f0c-448a-bbee-6688b8fa2bc5 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:42:01.036342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:01.036342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:01.036342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:01.036342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:01.036342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:01.036342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:01.036342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:01.036342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:01.036342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:01.036342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:01.036342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:01.036342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:01.036342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:01.036342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:01.036342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:01.037250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:01.037250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:01.037250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:01.037250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:01.037250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:01.037250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:01.037250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:01.037250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:01.037250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:01.037250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:01.037250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:01.037250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:01.037250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:01.037250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:01.037250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:01.037250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:01.038001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:01.038001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:01.038001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:01.038001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:01.038001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:01.038001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:01.038001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:01.038001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:01.038001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:01.038001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:01.038001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:42:01.038001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:42:01.038001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 57/118] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 18 00:42:00 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 369 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:01.039334 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb41144-7f0c-448a-bbee-6688b8fa2bc5 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:01.072185 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:01.133471 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7dd4f7c-42e0-470f-a66a-5b0e46200e77 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:01.174666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317f047d-11ad-40cb-9bd7-cb66be072f0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:01.175319 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317f047d-11ad-40cb-9bd7-cb66be072f0b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:01.175607 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317f047d-11ad-40cb-9bd7-cb66be072f0b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:01.175888 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317f047d-11ad-40cb-9bd7-cb66be072f0b None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:01.177128 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317f047d-11ad-40cb-9bd7-cb66be072f0b None admin] RBAC: Authorizing `identity:get_project(project_id=4c46ff2118f948a085f639b37068aa57)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:01.178666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317f047d-11ad-40cb-9bd7-cb66be072f0b None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:01.180364 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 62/119] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:00 2022] GET /identity/v3/projects/4c46ff2118f948a085f639b37068aa57 => generated 298 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:01.181803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb41144-7f0c-448a-bbee-6688b8fa2bc5 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:01.186698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317f047d-11ad-40cb-9bd7-cb66be072f0b None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:01.199162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:01.228980 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0876c716-e618-4a9f-96b3-b21f06ea73fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:01.229947 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0876c716-e618-4a9f-96b3-b21f06ea73fd None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:01.230365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0876c716-e618-4a9f-96b3-b21f06ea73fd None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:01.230926 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0876c716-e618-4a9f-96b3-b21f06ea73fd None admin] PATH_INFO: `/v3/roles/member` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:01.247186 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0876c716-e618-4a9f-96b3-b21f06ea73fd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:01.254249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-0876c716-e618-4a9f-96b3-b21f06ea73fd None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:01.254249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:01.254249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:01.254249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:01.254249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:01.254249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:01.254249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:01.254249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:01.254249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:01.254249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:01.254249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:01.254249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:01.254249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:01.254249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:01.254249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:01.254249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:01.255975 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:01.255975 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:01.255975 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:01.255975 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:01.255975 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:01.255975 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:01.255975 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:01.255975 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:01.255975 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:01.255975 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:01.255975 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:01.255975 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:01.255975 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:01.255975 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:01.255975 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:01.255975 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:01.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:01.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:01.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:01.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:01.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:01.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:01.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:01.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:01.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:01.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:01.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:01.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:42:01.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 58/120] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:01 2022] GET /identity/v3/roles/member => generated 84 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:01.260565 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0876c716-e618-4a9f-96b3-b21f06ea73fd None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:01.327819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317f047d-11ad-40cb-9bd7-cb66be072f0b None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:01.384132 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17fc135c-d674-4d61-ac01-27664dca07f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:01.385004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17fc135c-d674-4d61-ac01-27664dca07f9 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:01.385474 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17fc135c-d674-4d61-ac01-27664dca07f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:01.385785 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:01.385953 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17fc135c-d674-4d61-ac01-27664dca07f9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:01.422435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0876c716-e618-4a9f-96b3-b21f06ea73fd None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:01.480195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5121d86d-122f-4f9a-a699-41cba3cf8a50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:01.481004 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5121d86d-122f-4f9a-a699-41cba3cf8a50 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:01.481468 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5121d86d-122f-4f9a-a699-41cba3cf8a50 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:01.482504 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5121d86d-122f-4f9a-a699-41cba3cf8a50 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:01.500991 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5121d86d-122f-4f9a-a699-41cba3cf8a50 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:01.503072 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17fc135c-d674-4d61-ac01-27664dca07f9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:01.505761 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17fc135c-d674-4d61-ac01-27664dca07f9 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:01.511038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-5121d86d-122f-4f9a-a699-41cba3cf8a50 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:01.511038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:01.511038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:01.511038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:01.511038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:01.511038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:01.511038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:01.511038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:01.511038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:01.511038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:01.511038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:01.511038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:01.511038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:01.511038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:01.511038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:01.511038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:01.517402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:01.517402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:01.517402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:01.517402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:01.517402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:01.517402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:01.517402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:01.517402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:01.517402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:01.517402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:01.517402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:01.517402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:01.517402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:01.517402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:01.517402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:01.517402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:01.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:01.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:01.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:01.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:01.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:01.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:01.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:01.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:01.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:01.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:01.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:01.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:42:01.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 59/121] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:01 2022] GET /identity/v3/roles/member => generated 84 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:01.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5121d86d-122f-4f9a-a699-41cba3cf8a50 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:01.531009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:01.691474 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 63/122] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:42:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 506 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:42:01.691474 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17fc135c-d674-4d61-ac01-27664dca07f9 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:01.691734 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5121d86d-122f-4f9a-a699-41cba3cf8a50 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:01.745175 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc48177-63aa-4fbd-a76a-c45bb149824d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:01.746185 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc48177-63aa-4fbd-a76a-c45bb149824d None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:01.746690 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc48177-63aa-4fbd-a76a-c45bb149824d None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:01.747140 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc48177-63aa-4fbd-a76a-c45bb149824d None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:01.748326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc48177-63aa-4fbd-a76a-c45bb149824d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:01.748830 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc48177-63aa-4fbd-a76a-c45bb149824d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:01.751101 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc48177-63aa-4fbd-a76a-c45bb149824d None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:01.765463 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 60/123] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:42:01 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:01.766788 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc48177-63aa-4fbd-a76a-c45bb149824d None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:01.774512 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:01.871799 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17fc135c-d674-4d61-ac01-27664dca07f9 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:01.889623 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:01.935724 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452e0d44-2e29-46c1-ad46-c45b3263d48a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:01.936856 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452e0d44-2e29-46c1-ad46-c45b3263d48a None admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:01.937308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452e0d44-2e29-46c1-ad46-c45b3263d48a None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:01.937617 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc48177-63aa-4fbd-a76a-c45bb149824d None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:01.937798 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452e0d44-2e29-46c1-ad46-c45b3263d48a None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57/users/6f5558a706da458791737ed054e51874/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:01.941572 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452e0d44-2e29-46c1-ad46-c45b3263d48a None admin] RBAC: Authorizing `identity:create_grant(project_id=4c46ff2118f948a085f639b37068aa57, user_id=6f5558a706da458791737ed054e51874, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:01.943861 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452e0d44-2e29-46c1-ad46-c45b3263d48a None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:01.974509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 64/124] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:42:01 2022] PUT /identity/v3/projects/4c46ff2118f948a085f639b37068aa57/users/6f5558a706da458791737ed054e51874/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:42:01.974509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452e0d44-2e29-46c1-ad46-c45b3263d48a None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:02.015971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ffb660-a0d0-43bc-ba35-283ff584e042 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:02.017963 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ffb660-a0d0-43bc-ba35-283ff584e042 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:02.018623 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ffb660-a0d0-43bc-ba35-283ff584e042 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:02.019159 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ffb660-a0d0-43bc-ba35-283ff584e042 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:02.020881 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ffb660-a0d0-43bc-ba35-283ff584e042 None admin] RBAC: Adding query filter params (name=member) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:02.022029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ffb660-a0d0-43bc-ba35-283ff584e042 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:02.025054 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ffb660-a0d0-43bc-ba35-283ff584e042 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:02.046343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 61/125] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:01 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:02.048005 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ffb660-a0d0-43bc-ba35-283ff584e042 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:02.143321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452e0d44-2e29-46c1-ad46-c45b3263d48a None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:02.180119 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:02.194863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39ddcd6-e8f0-4978-af6d-3ceb3c98c974 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:02.195899 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39ddcd6-e8f0-4978-af6d-3ceb3c98c974 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:02.196489 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39ddcd6-e8f0-4978-af6d-3ceb3c98c974 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:02.196977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39ddcd6-e8f0-4978-af6d-3ceb3c98c974 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:02.198329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39ddcd6-e8f0-4978-af6d-3ceb3c98c974 None admin] RBAC: Adding query filter params (name=member) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:02.198906 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39ddcd6-e8f0-4978-af6d-3ceb3c98c974 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:02.202113 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39ddcd6-e8f0-4978-af6d-3ceb3c98c974 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:02.218674 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 65/126] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:01 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:02.219768 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39ddcd6-e8f0-4978-af6d-3ceb3c98c974 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:02.258919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ffb660-a0d0-43bc-ba35-283ff584e042 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:02.284119 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:02.310964 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338dc2e7-06c5-4564-913c-585c64bed62b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:02.312481 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338dc2e7-06c5-4564-913c-585c64bed62b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:02.312868 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338dc2e7-06c5-4564-913c-585c64bed62b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:02.313300 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338dc2e7-06c5-4564-913c-585c64bed62b None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:02.314783 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338dc2e7-06c5-4564-913c-585c64bed62b None admin] RBAC: Authorizing `identity:get_project(project_id=4c46ff2118f948a085f639b37068aa57)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:02.317069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338dc2e7-06c5-4564-913c-585c64bed62b None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:02.319639 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 62/127] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:02 2022] GET /identity/v3/projects/4c46ff2118f948a085f639b37068aa57 => generated 298 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:02.320563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338dc2e7-06c5-4564-913c-585c64bed62b None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:02.434200 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39ddcd6-e8f0-4978-af6d-3ceb3c98c974 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:02.434943 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338dc2e7-06c5-4564-913c-585c64bed62b None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:02.467020 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:02.471934 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ac31a0-4f6d-42fa-b6c3-c88708317e06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:02.472755 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ac31a0-4f6d-42fa-b6c3-c88708317e06 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:02.473009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ac31a0-4f6d-42fa-b6c3-c88708317e06 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:02.473275 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ac31a0-4f6d-42fa-b6c3-c88708317e06 None admin] PATH_INFO: `/v3/users/796c8ea379cf4e7c9570bc2fdba53211` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:02.474675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ac31a0-4f6d-42fa-b6c3-c88708317e06 None admin] RBAC: Authorizing `identity:get_user(user_id=796c8ea379cf4e7c9570bc2fdba53211)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:02.476281 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ac31a0-4f6d-42fa-b6c3-c88708317e06 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:02.477978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 63/128] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:02 2022] GET /identity/v3/users/796c8ea379cf4e7c9570bc2fdba53211 => generated 275 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:02.478968 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ac31a0-4f6d-42fa-b6c3-c88708317e06 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:02.505840 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d71d26-c9f7-4223-a083-50045d86f88a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:02.507762 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d71d26-c9f7-4223-a083-50045d86f88a None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:02.508602 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d71d26-c9f7-4223-a083-50045d86f88a None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:02.509567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d71d26-c9f7-4223-a083-50045d86f88a None admin] PATH_INFO: `/v3/users/796c8ea379cf4e7c9570bc2fdba53211` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:02.512203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d71d26-c9f7-4223-a083-50045d86f88a None admin] RBAC: Authorizing `identity:get_user(user_id=796c8ea379cf4e7c9570bc2fdba53211)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:02.515776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d71d26-c9f7-4223-a083-50045d86f88a None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:02.519446 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 66/129] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:02 2022] GET /identity/v3/users/796c8ea379cf4e7c9570bc2fdba53211 => generated 275 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:02.535155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d71d26-c9f7-4223-a083-50045d86f88a None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:02.598287 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ac31a0-4f6d-42fa-b6c3-c88708317e06 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:02.637159 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c74020-9f5d-4724-a2ce-c9ee04cb7d39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:02.637808 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c74020-9f5d-4724-a2ce-c9ee04cb7d39 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:02.638100 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c74020-9f5d-4724-a2ce-c9ee04cb7d39 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:02.638516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c74020-9f5d-4724-a2ce-c9ee04cb7d39 None admin] PATH_INFO: `/v3/users/796c8ea379cf4e7c9570bc2fdba53211` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:02.640457 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c74020-9f5d-4724-a2ce-c9ee04cb7d39 None admin] RBAC: Authorizing `identity:get_user(user_id=796c8ea379cf4e7c9570bc2fdba53211)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:02.642512 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c74020-9f5d-4724-a2ce-c9ee04cb7d39 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:02.644705 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 64/130] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:02 2022] GET /identity/v3/users/796c8ea379cf4e7c9570bc2fdba53211 => generated 275 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:02.649283 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c74020-9f5d-4724-a2ce-c9ee04cb7d39 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:02.701112 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:02.706843 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d71d26-c9f7-4223-a083-50045d86f88a None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:02.733695 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:02.748546 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edee5dc-ebce-44c8-b630-fc8fc840f759 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:02.749234 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edee5dc-ebce-44c8-b630-fc8fc840f759 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:02.749524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edee5dc-ebce-44c8-b630-fc8fc840f759 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:02.749804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edee5dc-ebce-44c8-b630-fc8fc840f759 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:02.750581 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0edee5dc-ebce-44c8-b630-fc8fc840f759 None admin] RBAC: Adding query filter params (role.id=a928285094b34996a609520c255b4c5d, scope.project.id=4c46ff2118f948a085f639b37068aa57, user.id=42fc7c5de9eb4a7cb41e4c636c8148e3) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:02.750900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0edee5dc-ebce-44c8-b630-fc8fc840f759 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:02.752411 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0edee5dc-ebce-44c8-b630-fc8fc840f759 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:02.759134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 67/131] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:42:02 2022] GET /identity/v3/role_assignments?user.id=42fc7c5de9eb4a7cb41e4c636c8148e3&scope.project.id=4c46ff2118f948a085f639b37068aa57&role.id=a928285094b34996a609520c255b4c5d => generated 259 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:02.760011 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edee5dc-ebce-44c8-b630-fc8fc840f759 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:02.814162 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c74020-9f5d-4724-a2ce-c9ee04cb7d39 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:02.873572 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba36626-b8bd-4a05-97ca-adb80a0386d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:02.874517 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba36626-b8bd-4a05-97ca-adb80a0386d5 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:02.874697 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba36626-b8bd-4a05-97ca-adb80a0386d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:02.874960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba36626-b8bd-4a05-97ca-adb80a0386d5 None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:02.876539 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba36626-b8bd-4a05-97ca-adb80a0386d5 None admin] RBAC: Authorizing `identity:get_project(project_id=4c46ff2118f948a085f639b37068aa57)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:02.878069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba36626-b8bd-4a05-97ca-adb80a0386d5 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:02.880124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 65/132] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:02 2022] GET /identity/v3/projects/4c46ff2118f948a085f639b37068aa57 => generated 298 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:02.881453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2406ea1e-1a74-4e4c-9ea7-7e91db213aaa None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:02.881746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2406ea1e-1a74-4e4c-9ea7-7e91db213aaa None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:02.882025 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2406ea1e-1a74-4e4c-9ea7-7e91db213aaa None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:02.883922 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 66/133] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:02.884716 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2406ea1e-1a74-4e4c-9ea7-7e91db213aaa None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:02.950358 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edee5dc-ebce-44c8-b630-fc8fc840f759 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:03.013039 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2406ea1e-1a74-4e4c-9ea7-7e91db213aaa None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:03.020209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6a96b1-e17f-45e5-9645-dcd89a214b21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:03.022376 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6a96b1-e17f-45e5-9645-dcd89a214b21 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:03.023218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6a96b1-e17f-45e5-9645-dcd89a214b21 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:03.024146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6a96b1-e17f-45e5-9645-dcd89a214b21 None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:03.026582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6a96b1-e17f-45e5-9645-dcd89a214b21 None admin] RBAC: Authorizing `identity:get_project(project_id=4c46ff2118f948a085f639b37068aa57)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:03.030077 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6a96b1-e17f-45e5-9645-dcd89a214b21 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:03.038662 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 68/134] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:02 2022] GET /identity/v3/projects/4c46ff2118f948a085f639b37068aa57 => generated 298 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:03.038662 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939629d9-e194-4ff6-bde0-748db71b3591 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:03.038662 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939629d9-e194-4ff6-bde0-748db71b3591 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:03.038662 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939629d9-e194-4ff6-bde0-748db71b3591 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:03.062202 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b36e20-4a89-4e46-9bcd-e3ca8c433a93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:03.063060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b36e20-4a89-4e46-9bcd-e3ca8c433a93 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:03.063448 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b36e20-4a89-4e46-9bcd-e3ca8c433a93 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:03.063836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b36e20-4a89-4e46-9bcd-e3ca8c433a93 None admin] PATH_INFO: `/v3/projects/c3a25b7f689e4e21b2f61ed886a2a2d2` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:03.065303 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b36e20-4a89-4e46-9bcd-e3ca8c433a93 None admin] RBAC: Authorizing `identity:get_project(project_id=c3a25b7f689e4e21b2f61ed886a2a2d2)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:03.067848 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b36e20-4a89-4e46-9bcd-e3ca8c433a93 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:03.070594 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 67/135] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:02 2022] GET /identity/v3/projects/c3a25b7f689e4e21b2f61ed886a2a2d2 => generated 312 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:03.071747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b36e20-4a89-4e46-9bcd-e3ca8c433a93 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:03.076995 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:03.078994 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-939629d9-e194-4ff6-bde0-748db71b3591 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:03.081689 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-939629d9-e194-4ff6-bde0-748db71b3591 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:03.129178 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:03.192330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 69/136] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:03.192330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939629d9-e194-4ff6-bde0-748db71b3591 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:03.259042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b36e20-4a89-4e46-9bcd-e3ca8c433a93 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:03.319376 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939629d9-e194-4ff6-bde0-748db71b3591 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:03.342057 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa20ea0-70a2-4559-a8a6-5e24d24013f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:03.343650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa20ea0-70a2-4559-a8a6-5e24d24013f7 None admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:03.344332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa20ea0-70a2-4559-a8a6-5e24d24013f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:03.345033 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa20ea0-70a2-4559-a8a6-5e24d24013f7 None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57/users/796c8ea379cf4e7c9570bc2fdba53211/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:03.348863 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa20ea0-70a2-4559-a8a6-5e24d24013f7 None admin] RBAC: Authorizing `identity:create_grant(project_id=4c46ff2118f948a085f639b37068aa57, user_id=796c8ea379cf4e7c9570bc2fdba53211, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:03.352144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa20ea0-70a2-4559-a8a6-5e24d24013f7 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:03.360318 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9518efce-defb-42ac-bab0-fef0f38c47a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:03.361784 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9518efce-defb-42ac-bab0-fef0f38c47a1 None admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:03.361784 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9518efce-defb-42ac-bab0-fef0f38c47a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:03.361784 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9518efce-defb-42ac-bab0-fef0f38c47a1 None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57/users/796c8ea379cf4e7c9570bc2fdba53211/roles/cf934cf252ce42d5a0445d3e156348d1` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:03.365532 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9518efce-defb-42ac-bab0-fef0f38c47a1 None admin] RBAC: Authorizing `identity:create_grant(project_id=4c46ff2118f948a085f639b37068aa57, user_id=796c8ea379cf4e7c9570bc2fdba53211, role_id=cf934cf252ce42d5a0445d3e156348d1)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:03.366680 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9518efce-defb-42ac-bab0-fef0f38c47a1 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:03.384989 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 68/137] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:42:03 2022] PUT /identity/v3/projects/4c46ff2118f948a085f639b37068aa57/users/796c8ea379cf4e7c9570bc2fdba53211/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:42:03.387352 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa20ea0-70a2-4559-a8a6-5e24d24013f7 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:03.398290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 70/138] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:42:03 2022] PUT /identity/v3/projects/4c46ff2118f948a085f639b37068aa57/users/796c8ea379cf4e7c9570bc2fdba53211/roles/cf934cf252ce42d5a0445d3e156348d1 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:42:03.514036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:03.588353 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa20ea0-70a2-4559-a8a6-5e24d24013f7 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:03.648816 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad87ec5-d59f-4d2f-81e3-45d50e95105f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:03.649832 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad87ec5-d59f-4d2f-81e3-45d50e95105f None admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:03.650257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad87ec5-d59f-4d2f-81e3-45d50e95105f None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:03.650720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad87ec5-d59f-4d2f-81e3-45d50e95105f None admin] PATH_INFO: `/v3/projects/c3a25b7f689e4e21b2f61ed886a2a2d2/users/796c8ea379cf4e7c9570bc2fdba53211/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:03.654434 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad87ec5-d59f-4d2f-81e3-45d50e95105f None admin] RBAC: Authorizing `identity:create_grant(project_id=c3a25b7f689e4e21b2f61ed886a2a2d2, user_id=796c8ea379cf4e7c9570bc2fdba53211, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:03.657178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad87ec5-d59f-4d2f-81e3-45d50e95105f None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:03.680583 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 69/139] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:42:03 2022] PUT /identity/v3/projects/c3a25b7f689e4e21b2f61ed886a2a2d2/users/796c8ea379cf4e7c9570bc2fdba53211/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:42:03.812331 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9518efce-defb-42ac-bab0-fef0f38c47a1 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:03.841524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:03.945807 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9518efce-defb-42ac-bab0-fef0f38c47a1 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:03.989498 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf4534f-182e-4ab6-9a67-0290c8480b78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:03.990327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf4534f-182e-4ab6-9a67-0290c8480b78 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:03.990762 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf4534f-182e-4ab6-9a67-0290c8480b78 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:03.991153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf4534f-182e-4ab6-9a67-0290c8480b78 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:03.992796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf4534f-182e-4ab6-9a67-0290c8480b78 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:03.994449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf4534f-182e-4ab6-9a67-0290c8480b78 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:03.996488 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 71/140] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:42:03 2022] GET /identity/v3/domains/default => generated 199 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:04.003522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad87ec5-d59f-4d2f-81e3-45d50e95105f None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:04.138940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad87ec5-d59f-4d2f-81e3-45d50e95105f None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:04.171935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:04.189363 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2605976-c96c-4c28-855b-ac6fd809749a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:04.190170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2605976-c96c-4c28-855b-ac6fd809749a None admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:04.190588 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2605976-c96c-4c28-855b-ac6fd809749a None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:04.190969 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2605976-c96c-4c28-855b-ac6fd809749a None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:04.191764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2605976-c96c-4c28-855b-ac6fd809749a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:04.193595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2605976-c96c-4c28-855b-ac6fd809749a None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:04.229621 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 70/141] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 18 00:42:04 2022] POST /identity/v3/users => generated 283 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:42:06.101392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ba85c8-be92-487b-a363-41c171e589b2 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:06.102102 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ba85c8-be92-487b-a363-41c171e589b2 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:06.102102 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ba85c8-be92-487b-a363-41c171e589b2 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:06.103822 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 72/142] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:06.112212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d1f5ec-82c5-40aa-b848-c7105a4ccb07 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:06.112909 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d1f5ec-82c5-40aa-b848-c7105a4ccb07 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:06.112909 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d1f5ec-82c5-40aa-b848-c7105a4ccb07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:06.141499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-b6d1f5ec-82c5-40aa-b848-c7105a4ccb07 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:06.142904 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-b6d1f5ec-82c5-40aa-b848-c7105a4ccb07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:06.153437 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c74a88b-022a-4731-9394-51d43a3c575b None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:06.154067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c74a88b-022a-4731-9394-51d43a3c575b None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:06.154582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c74a88b-022a-4731-9394-51d43a3c575b None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:06.157130 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 73/143] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:06 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:06.166549 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bccca2-bd40-4f31-be25-81bad50c0755 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:06.167038 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bccca2-bd40-4f31-be25-81bad50c0755 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:06.167474 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bccca2-bd40-4f31-be25-81bad50c0755 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:06.202706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-16bccca2-bd40-4f31-be25-81bad50c0755 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:06.204832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-16bccca2-bd40-4f31-be25-81bad50c0755 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:06.303505 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 71/144] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:06.391671 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 74/145] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:06.946711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d1f5ec-82c5-40aa-b848-c7105a4ccb07 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:07.085231 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d1f5ec-82c5-40aa-b848-c7105a4ccb07 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:07.107116 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:07.139456 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8446e7-7eae-4173-b94c-979617806b97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:07.140444 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8446e7-7eae-4173-b94c-979617806b97 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:07.140444 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8446e7-7eae-4173-b94c-979617806b97 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:07.140716 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8446e7-7eae-4173-b94c-979617806b97 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:07.159476 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb8446e7-7eae-4173-b94c-979617806b97 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:07.164597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-bb8446e7-7eae-4173-b94c-979617806b97 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 18 00:42:07.164597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:07.164597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:07.164597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:07.164597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:07.164597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:07.164597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:07.164597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:07.164597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:07.164597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:07.164597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:07.164597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:07.164597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:07.164597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:07.164597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:07.164597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:07.166132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:07.166132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:07.166132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:07.166132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:07.166132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:07.166132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:07.166132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:07.166132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:07.166132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:07.166132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:07.166132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:07.166132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:07.166132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:07.166132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:07.166132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:07.166132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:07.167570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:07.167570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:07.167570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:07.167570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:07.167570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:07.167570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:07.167570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:07.167570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:07.167570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:07.167570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:07.167570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 18 00:42:07.167570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:42:07.167570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 72/146] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:06 2022] GET /identity/v3/roles/reader => generated 84 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:07.171872 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bccca2-bd40-4f31-be25-81bad50c0755 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:07.197942 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:07.306916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bccca2-bd40-4f31-be25-81bad50c0755 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:07.353794 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59e5952-1005-4f3e-be13-877f6b8570d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:07.354606 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59e5952-1005-4f3e-be13-877f6b8570d5 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:07.354874 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59e5952-1005-4f3e-be13-877f6b8570d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:07.355208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59e5952-1005-4f3e-be13-877f6b8570d5 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:07.355923 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59e5952-1005-4f3e-be13-877f6b8570d5 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:07.356224 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59e5952-1005-4f3e-be13-877f6b8570d5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:07.357795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59e5952-1005-4f3e-be13-877f6b8570d5 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:07.367515 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 75/147] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:07 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:07.373792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8446e7-7eae-4173-b94c-979617806b97 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:07.481300 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59e5952-1005-4f3e-be13-877f6b8570d5 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:07.524585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8446e7-7eae-4173-b94c-979617806b97 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:07.609921 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4154603d-1194-491f-9c64-17a2998eb169 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:07.610633 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4154603d-1194-491f-9c64-17a2998eb169 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:07.610913 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4154603d-1194-491f-9c64-17a2998eb169 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:07.611186 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4154603d-1194-491f-9c64-17a2998eb169 None admin] PATH_INFO: `/v3/users/42fc7c5de9eb4a7cb41e4c636c8148e3` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:07.615964 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4154603d-1194-491f-9c64-17a2998eb169 None admin] RBAC: Authorizing `identity:get_user(user_id=42fc7c5de9eb4a7cb41e4c636c8148e3)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:07.617659 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4154603d-1194-491f-9c64-17a2998eb169 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:07.619946 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 73/148] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:07 2022] GET /identity/v3/users/42fc7c5de9eb4a7cb41e4c636c8148e3 => generated 289 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:07.628116 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4154603d-1194-491f-9c64-17a2998eb169 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:07.697484 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:07.714216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:07.762104 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59e5952-1005-4f3e-be13-877f6b8570d5 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:07.878913 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4154603d-1194-491f-9c64-17a2998eb169 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:07.919784 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99e50af-ff3b-46f8-b07f-db0933a4c1df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:07.920851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99e50af-ff3b-46f8-b07f-db0933a4c1df None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:07.921328 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99e50af-ff3b-46f8-b07f-db0933a4c1df None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:07.921811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99e50af-ff3b-46f8-b07f-db0933a4c1df None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:07.964710 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f6dea4-7a84-4b85-bc7f-fbafeec3f46f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:07.965382 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f6dea4-7a84-4b85-bc7f-fbafeec3f46f None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:07.965656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f6dea4-7a84-4b85-bc7f-fbafeec3f46f None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:07.965928 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f6dea4-7a84-4b85-bc7f-fbafeec3f46f None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:07.967853 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f6dea4-7a84-4b85-bc7f-fbafeec3f46f None admin] RBAC: Authorizing `identity:get_project(project_id=4c46ff2118f948a085f639b37068aa57)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:07.968107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99e50af-ff3b-46f8-b07f-db0933a4c1df None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:07.969341 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f6dea4-7a84-4b85-bc7f-fbafeec3f46f None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:07.971076 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 74/149] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:07 2022] GET /identity/v3/projects/4c46ff2118f948a085f639b37068aa57 => generated 298 bytes in 344 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:07.985852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-b99e50af-ff3b-46f8-b07f-db0933a4c1df None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:42:07.985852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:07.985852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:07.985852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:07.985852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:07.985852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:07.985852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:07.985852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:07.985852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:07.985852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:07.985852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:07.985852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:07.985852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:07.985852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:07.985852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:07.985852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:07.986803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:07.986803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:07.986803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:07.986803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:07.986803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:07.986803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:07.986803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:07.986803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:07.986803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:07.986803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:07.986803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:07.986803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:07.986803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:07.986803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:07.986803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:07.986803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:07.987585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:07.987585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:07.987585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:07.987585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:07.987585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:07.987585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:07.987585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:07.987585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:07.987585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:07.987585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:07.987585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:42:07.987585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:42:07.988228 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f6dea4-7a84-4b85-bc7f-fbafeec3f46f None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:07.988852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 76/150] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:42:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 508 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:42:07.999616 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99e50af-ff3b-46f8-b07f-db0933a4c1df None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:08.221926 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f6dea4-7a84-4b85-bc7f-fbafeec3f46f None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:08.234839 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:08.299137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5cd3e0-ea2f-48d5-80d6-e396751924e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:08.300080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5cd3e0-ea2f-48d5-80d6-e396751924e0 None admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:08.300357 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5cd3e0-ea2f-48d5-80d6-e396751924e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:08.300636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5cd3e0-ea2f-48d5-80d6-e396751924e0 None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57/users/42fc7c5de9eb4a7cb41e4c636c8148e3/roles/a928285094b34996a609520c255b4c5d` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:08.309056 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5cd3e0-ea2f-48d5-80d6-e396751924e0 None admin] RBAC: Authorizing `identity:create_grant(project_id=4c46ff2118f948a085f639b37068aa57, user_id=42fc7c5de9eb4a7cb41e4c636c8148e3, role_id=a928285094b34996a609520c255b4c5d)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:08.310762 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5cd3e0-ea2f-48d5-80d6-e396751924e0 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:08.344203 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 75/151] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:42:07 2022] PUT /identity/v3/projects/4c46ff2118f948a085f639b37068aa57/users/42fc7c5de9eb4a7cb41e4c636c8148e3/roles/a928285094b34996a609520c255b4c5d => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:42:08.369659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99e50af-ff3b-46f8-b07f-db0933a4c1df None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:08.453493 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5a4836-3c4b-4021-ad03-e09c617a6db1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:08.454566 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5a4836-3c4b-4021-ad03-e09c617a6db1 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:08.455013 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5a4836-3c4b-4021-ad03-e09c617a6db1 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:08.455429 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5a4836-3c4b-4021-ad03-e09c617a6db1 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:08.456562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b5a4836-3c4b-4021-ad03-e09c617a6db1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:08.457093 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b5a4836-3c4b-4021-ad03-e09c617a6db1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:08.459630 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b5a4836-3c4b-4021-ad03-e09c617a6db1 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:08.479680 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 77/152] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 482 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:08.488470 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5cd3e0-ea2f-48d5-80d6-e396751924e0 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:08.655016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5cd3e0-ea2f-48d5-80d6-e396751924e0 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:08.690491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249d9ad8-6660-488a-802b-5ebc5e7ccbad None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:08.690798 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249d9ad8-6660-488a-802b-5ebc5e7ccbad None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:08.691082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249d9ad8-6660-488a-802b-5ebc5e7ccbad None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:08.692494 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 78/153] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:42:08.711729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b50b2b8-afd9-4617-a2b7-26b32cb22f6e None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:08.712032 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b50b2b8-afd9-4617-a2b7-26b32cb22f6e None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:08.712292 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b50b2b8-afd9-4617-a2b7-26b32cb22f6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:08.713566 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b298a9-b1c7-4825-9b63-171d8e44d815 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:08.714226 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b298a9-b1c7-4825-9b63-171d8e44d815 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:08.714528 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b298a9-b1c7-4825-9b63-171d8e44d815 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:08.714803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b298a9-b1c7-4825-9b63-171d8e44d815 None admin] PATH_INFO: `/v3/users/6f5558a706da458791737ed054e51874` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:08.719351 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b298a9-b1c7-4825-9b63-171d8e44d815 None admin] RBAC: Authorizing `identity:get_user(user_id=6f5558a706da458791737ed054e51874)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:08.720924 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b298a9-b1c7-4825-9b63-171d8e44d815 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:08.722786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 76/154] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:08 2022] GET /identity/v3/users/6f5558a706da458791737ed054e51874 => generated 247 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:08.730051 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:08.732247 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b298a9-b1c7-4825-9b63-171d8e44d815 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:08.755705 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-2b50b2b8-afd9-4617-a2b7-26b32cb22f6e None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:08.756954 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-2b50b2b8-afd9-4617-a2b7-26b32cb22f6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:08.850925 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:08.921691 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 79/155] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:42:08.923267 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a654e98-7357-4607-9bb9-aff2984c1a44 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:08.923743 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a654e98-7357-4607-9bb9-aff2984c1a44 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:08.924023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a654e98-7357-4607-9bb9-aff2984c1a44 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:08.925862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 80/156] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:08.931811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77302133-bade-4e31-806b-81472a90d2ac None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:08.932142 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77302133-bade-4e31-806b-81472a90d2ac None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:08.932377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77302133-bade-4e31-806b-81472a90d2ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:08.960307 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-77302133-bade-4e31-806b-81472a90d2ac None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:08.961601 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-77302133-bade-4e31-806b-81472a90d2ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:09.026878 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b298a9-b1c7-4825-9b63-171d8e44d815 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:09.109913 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 81/157] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:09.113874 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2357b259-3aa3-489e-a9d2-5570b93493dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:09.114975 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2357b259-3aa3-489e-a9d2-5570b93493dc None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:09.115465 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2357b259-3aa3-489e-a9d2-5570b93493dc None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:09.115940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2357b259-3aa3-489e-a9d2-5570b93493dc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:09.285912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2357b259-3aa3-489e-a9d2-5570b93493dc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:09.293098 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2357b259-3aa3-489e-a9d2-5570b93493dc None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:09.365744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:09.491411 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 77/158] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:42:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 760 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:42:09.500855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77302133-bade-4e31-806b-81472a90d2ac None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:09.613475 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:09.671032 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77302133-bade-4e31-806b-81472a90d2ac None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:09.675826 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2357b259-3aa3-489e-a9d2-5570b93493dc None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:09.722329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1eea33c-8b01-478e-8081-49429b1ce0e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:09.723041 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1eea33c-8b01-478e-8081-49429b1ce0e7 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:09.723323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1eea33c-8b01-478e-8081-49429b1ce0e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:09.723600 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1eea33c-8b01-478e-8081-49429b1ce0e7 None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:09.725108 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1eea33c-8b01-478e-8081-49429b1ce0e7 None admin] RBAC: Authorizing `identity:get_project(project_id=4c46ff2118f948a085f639b37068aa57)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:09.726505 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1eea33c-8b01-478e-8081-49429b1ce0e7 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:09.728423 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 82/159] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:09 2022] GET /identity/v3/projects/4c46ff2118f948a085f639b37068aa57 => generated 298 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:09.735351 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1eea33c-8b01-478e-8081-49429b1ce0e7 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:09.925126 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2357b259-3aa3-489e-a9d2-5570b93493dc None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:09.954241 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1eea33c-8b01-478e-8081-49429b1ce0e7 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:09.994543 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b313db70-76da-4b74-9250-e29abf93f02a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:09.995207 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b313db70-76da-4b74-9250-e29abf93f02a None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:09.995487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b313db70-76da-4b74-9250-e29abf93f02a None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:09.995763 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b313db70-76da-4b74-9250-e29abf93f02a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:09.999441 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:10.015787 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:10.023716 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b313db70-76da-4b74-9250-e29abf93f02a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:10.028860 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-b313db70-76da-4b74-9250-e29abf93f02a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:10.028860 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:10.028860 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:10.028860 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:10.028860 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:10.028860 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:10.028860 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:10.028860 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:10.028860 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:10.028860 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:10.028860 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:10.028860 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:10.028860 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:10.028860 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:10.028860 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:10.028860 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:10.030356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:10.030356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:10.030356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:10.030356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:10.030356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:10.030356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:10.030356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:10.030356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:10.030356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:10.030356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:10.030356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:10.030356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:10.030356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:10.030356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:10.030356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:10.030356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:10.031871 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:10.031871 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:10.031871 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:10.031871 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:10.031871 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:10.031871 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:10.031871 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:10.031871 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:10.031871 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:10.031871 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:10.031871 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:10.031871 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:42:10.033130 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7911f99e-805b-4289-89fb-a524c3e6b9bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:10.033130 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7911f99e-805b-4289-89fb-a524c3e6b9bd None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:10.033397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7911f99e-805b-4289-89fb-a524c3e6b9bd None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:10.033532 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7911f99e-805b-4289-89fb-a524c3e6b9bd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:10.034575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7911f99e-805b-4289-89fb-a524c3e6b9bd None admin] RBAC: Adding query filter params (role.id=1e04c9fea65244d28e1e59fc4c64f32f, scope.project.id=4c46ff2118f948a085f639b37068aa57, user.id=6f5558a706da458791737ed054e51874) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:10.034575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7911f99e-805b-4289-89fb-a524c3e6b9bd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:10.038559 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7911f99e-805b-4289-89fb-a524c3e6b9bd None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:10.038876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 78/160] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:09 2022] GET /identity/v3/roles/member => generated 84 bytes in 364 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:10.039654 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530f200b-4a01-4bca-bfd7-d19c7b6ea1b9 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:10.039938 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530f200b-4a01-4bca-bfd7-d19c7b6ea1b9 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:10.040220 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530f200b-4a01-4bca-bfd7-d19c7b6ea1b9 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:10.045306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 79/161] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:10 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:10.050167 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df37a53f-a80d-4be0-a927-99590f130163 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:10.050525 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df37a53f-a80d-4be0-a927-99590f130163 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:10.050781 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df37a53f-a80d-4be0-a927-99590f130163 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:10.052994 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 83/162] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:42:09 2022] GET /identity/v3/role_assignments?user.id=6f5558a706da458791737ed054e51874&scope.project.id=4c46ff2118f948a085f639b37068aa57&role.id=1e04c9fea65244d28e1e59fc4c64f32f => generated 608 bytes in 318 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:10.053863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7911f99e-805b-4289-89fb-a524c3e6b9bd None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:10.055981 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 80/163] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:10 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:10.056777 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df37a53f-a80d-4be0-a927-99590f130163 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:10.319610 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7911f99e-805b-4289-89fb-a524c3e6b9bd None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:10.334505 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df37a53f-a80d-4be0-a927-99590f130163 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:10.376664 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a221b3-3615-4651-afee-c11079c91b10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:10.377370 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a221b3-3615-4651-afee-c11079c91b10 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:10.377595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a221b3-3615-4651-afee-c11079c91b10 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:10.377866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a221b3-3615-4651-afee-c11079c91b10 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:10.379072 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a221b3-3615-4651-afee-c11079c91b10 None admin] RBAC: Adding query filter params (name=member) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:10.379390 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a221b3-3615-4651-afee-c11079c91b10 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:10.381030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a221b3-3615-4651-afee-c11079c91b10 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:10.392249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 81/164] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:10 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 336 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:10.393412 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8134d80d-55fe-4d7a-a12a-b39fcd45d2c7 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:10.393703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8134d80d-55fe-4d7a-a12a-b39fcd45d2c7 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:10.393982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8134d80d-55fe-4d7a-a12a-b39fcd45d2c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:10.416409 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-8134d80d-55fe-4d7a-a12a-b39fcd45d2c7 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:10.417858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8134d80d-55fe-4d7a-a12a-b39fcd45d2c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:10.443805 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521f8822-1f2f-4df3-8176-5ab85f4579d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:10.444889 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521f8822-1f2f-4df3-8176-5ab85f4579d5 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:10.445383 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521f8822-1f2f-4df3-8176-5ab85f4579d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:10.445862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521f8822-1f2f-4df3-8176-5ab85f4579d5 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:10.483901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521f8822-1f2f-4df3-8176-5ab85f4579d5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:10.494151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-521f8822-1f2f-4df3-8176-5ab85f4579d5 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:42:10.494151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:10.494151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:10.494151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:10.494151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:10.494151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:10.494151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:10.494151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:10.494151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:10.494151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:10.494151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:10.494151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:10.494151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:10.494151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:10.494151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:10.494151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:10.495144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:10.495144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:10.495144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:10.495144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:10.495144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:10.495144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:10.495144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:10.495144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:10.495144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:10.495144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:10.495144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:10.495144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:10.495144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:10.495144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:10.495144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:10.495144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:10.495930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:10.495930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:10.495930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:10.495930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:10.495930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:10.495930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:10.495930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:10.495930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:10.495930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:10.495930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:10.495930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:42:10.495930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:42:10.501309 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 84/165] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 18 00:42:10 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 448 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:10.505611 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b12e3de-f2ac-46c0-a9f5-3cd987a2336e None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:10.514498 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b12e3de-f2ac-46c0-a9f5-3cd987a2336e None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:10.515219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b12e3de-f2ac-46c0-a9f5-3cd987a2336e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:10.533395 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:10.561346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-4b12e3de-f2ac-46c0-a9f5-3cd987a2336e None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:10.563337 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-4b12e3de-f2ac-46c0-a9f5-3cd987a2336e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:10.575165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 82/166] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:10.576590 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8134d80d-55fe-4d7a-a12a-b39fcd45d2c7 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:10.681519 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:10.715811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 85/167] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:10.716577 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b12e3de-f2ac-46c0-a9f5-3cd987a2336e None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:10.820749 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8134d80d-55fe-4d7a-a12a-b39fcd45d2c7 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:10.891636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5891bb-8825-4f71-8a30-2b2c1cdb93df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:10.892330 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5891bb-8825-4f71-8a30-2b2c1cdb93df None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:10.892588 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5891bb-8825-4f71-8a30-2b2c1cdb93df None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:10.892850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5891bb-8825-4f71-8a30-2b2c1cdb93df None admin] PATH_INFO: `/v3/users/796c8ea379cf4e7c9570bc2fdba53211` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:10.894285 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5891bb-8825-4f71-8a30-2b2c1cdb93df None admin] RBAC: Authorizing `identity:get_user(user_id=796c8ea379cf4e7c9570bc2fdba53211)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:10.895805 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5891bb-8825-4f71-8a30-2b2c1cdb93df None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:10.900129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 83/168] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:10 2022] GET /identity/v3/users/796c8ea379cf4e7c9570bc2fdba53211 => generated 275 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:10.902383 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e242fe-59fb-4df4-9392-c9e14e27f6ca None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:10.902691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e242fe-59fb-4df4-9392-c9e14e27f6ca None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:10.902965 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e242fe-59fb-4df4-9392-c9e14e27f6ca None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:10.904872 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 84/169] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:10.905631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e242fe-59fb-4df4-9392-c9e14e27f6ca None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:10.926836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b12e3de-f2ac-46c0-a9f5-3cd987a2336e None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:10.995342 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2614ec8-1dd5-4704-9ae6-91d3eff32fc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:10.995979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2614ec8-1dd5-4704-9ae6-91d3eff32fc4 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:10.996256 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2614ec8-1dd5-4704-9ae6-91d3eff32fc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:10.996524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2614ec8-1dd5-4704-9ae6-91d3eff32fc4 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:10.997218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2614ec8-1dd5-4704-9ae6-91d3eff32fc4 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:10.998067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2614ec8-1dd5-4704-9ae6-91d3eff32fc4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:10.999524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2614ec8-1dd5-4704-9ae6-91d3eff32fc4 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:11.019393 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 86/170] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:42:10 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:11.020704 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0780c26d-eafb-4bff-aa18-451c94b4aec0 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:11.021055 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0780c26d-eafb-4bff-aa18-451c94b4aec0 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:11.021262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0780c26d-eafb-4bff-aa18-451c94b4aec0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:11.036302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:11.060745 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-0780c26d-eafb-4bff-aa18-451c94b4aec0 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:11.061995 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-0780c26d-eafb-4bff-aa18-451c94b4aec0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:11.142651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e242fe-59fb-4df4-9392-c9e14e27f6ca None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:11.151665 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:11.228574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 87/171] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:11.230124 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0780c26d-eafb-4bff-aa18-451c94b4aec0 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:11.230418 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4642b923-a6e4-4699-add2-ea1344c9bdbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:11.231068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4642b923-a6e4-4699-add2-ea1344c9bdbd None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:11.231342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4642b923-a6e4-4699-add2-ea1344c9bdbd None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:11.231607 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4642b923-a6e4-4699-add2-ea1344c9bdbd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:11.364986 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4642b923-a6e4-4699-add2-ea1344c9bdbd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:11.366686 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4642b923-a6e4-4699-add2-ea1344c9bdbd None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:11.451286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0780c26d-eafb-4bff-aa18-451c94b4aec0 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:11.485636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:11.505469 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a2c33f-e49e-4fbc-8755-5e84a22470db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:11.506148 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a2c33f-e49e-4fbc-8755-5e84a22470db None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:11.506392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a2c33f-e49e-4fbc-8755-5e84a22470db None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:11.506697 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a2c33f-e49e-4fbc-8755-5e84a22470db None admin] PATH_INFO: `/v3/users/796c8ea379cf4e7c9570bc2fdba53211` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:11.507934 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a2c33f-e49e-4fbc-8755-5e84a22470db None admin] RBAC: Authorizing `identity:get_user(user_id=796c8ea379cf4e7c9570bc2fdba53211)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:11.511240 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a2c33f-e49e-4fbc-8755-5e84a22470db None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:11.518517 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 88/172] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:11 2022] GET /identity/v3/users/796c8ea379cf4e7c9570bc2fdba53211 => generated 275 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:11.518517 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a2c33f-e49e-4fbc-8755-5e84a22470db None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:11.569231 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:11.594764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 85/173] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:42:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 689 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:42:11.596642 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4642b923-a6e4-4699-add2-ea1344c9bdbd None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:11.660057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a2c33f-e49e-4fbc-8755-5e84a22470db None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:11.710041 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4642b923-a6e4-4699-add2-ea1344c9bdbd None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:11.728713 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37da64b0-3b24-4f31-b82f-f1c58e38305b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:11.729983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37da64b0-3b24-4f31-b82f-f1c58e38305b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:11.730503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37da64b0-3b24-4f31-b82f-f1c58e38305b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:11.731003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37da64b0-3b24-4f31-b82f-f1c58e38305b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:11.749324 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bd636c-ffc3-40ee-8274-6d3814148782 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:11.749971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bd636c-ffc3-40ee-8274-6d3814148782 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:11.750245 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bd636c-ffc3-40ee-8274-6d3814148782 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:11.750541 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bd636c-ffc3-40ee-8274-6d3814148782 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:11.752436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37da64b0-3b24-4f31-b82f-f1c58e38305b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:11.762559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9bd636c-ffc3-40ee-8274-6d3814148782 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:11.763332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-37da64b0-3b24-4f31-b82f-f1c58e38305b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:42:11.763332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:11.763332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:11.763332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:11.763332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:11.763332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:11.763332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:11.763332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:11.763332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:11.763332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:11.763332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:11.763332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:11.763332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:11.763332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:11.763332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:11.763332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:11.764199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:11.764199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:11.764199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:11.764199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:11.764199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:11.764199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:11.764199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:11.764199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:11.764199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:11.764199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:11.764199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:11.764199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:11.764199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:11.764199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:11.764199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:11.764199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:11.765009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:11.765009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:11.765009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:11.765009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:11.765009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:11.765009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:11.765009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:11.765009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:11.765009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:11.765009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:11.765009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:42:11.765009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:42:11.765678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 89/174] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:42:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:11.767210 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37da64b0-3b24-4f31-b82f-f1c58e38305b None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:11.769464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-e9bd636c-ffc3-40ee-8274-6d3814148782 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:11.769464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:11.769464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:11.769464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:11.769464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:11.769464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:11.769464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:11.769464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:11.769464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:11.769464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:11.769464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:11.769464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:11.769464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:11.769464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:11.769464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:11.769464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:11.771570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:11.771570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:11.771570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:11.771570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:11.771570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:11.771570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:11.771570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:11.771570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:11.771570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:11.771570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:11.771570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:11.771570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:11.771570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:11.771570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:11.771570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:11.771570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:11.772864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:11.772864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:11.772864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:11.772864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:11.772864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:11.772864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:11.772864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:11.772864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:11.772864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:11.772864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:11.772864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:11.772864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:42:11.772864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 86/175] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:11 2022] GET /identity/v3/roles/member => generated 84 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:11.780004 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640826a4-141d-4699-92b5-3239f378cd6b None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:11.780004 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640826a4-141d-4699-92b5-3239f378cd6b None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:11.780004 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640826a4-141d-4699-92b5-3239f378cd6b None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:11.780004 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 87/176] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:11 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:11.781153 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640826a4-141d-4699-92b5-3239f378cd6b None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:11.799969 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:11.928456 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640826a4-141d-4699-92b5-3239f378cd6b None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:11.953409 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37da64b0-3b24-4f31-b82f-f1c58e38305b None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:11.957511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:11.987251 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc7ff11-c31c-4f25-85ce-96e3a67c75f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:11.987826 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc7ff11-c31c-4f25-85ce-96e3a67c75f8 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:11.988110 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc7ff11-c31c-4f25-85ce-96e3a67c75f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:11.988367 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc7ff11-c31c-4f25-85ce-96e3a67c75f8 None admin] PATH_INFO: `/v3/projects/c3a25b7f689e4e21b2f61ed886a2a2d2` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:11.989677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc7ff11-c31c-4f25-85ce-96e3a67c75f8 None admin] RBAC: Authorizing `identity:get_project(project_id=c3a25b7f689e4e21b2f61ed886a2a2d2)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:11.991247 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc7ff11-c31c-4f25-85ce-96e3a67c75f8 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:12.001572 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 88/177] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:11 2022] GET /identity/v3/projects/c3a25b7f689e4e21b2f61ed886a2a2d2 => generated 312 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:12.001572 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc7ff11-c31c-4f25-85ce-96e3a67c75f8 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:12.017378 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c755d8-c14d-4b07-87d9-4f9ea2e5bb61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:12.018361 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c755d8-c14d-4b07-87d9-4f9ea2e5bb61 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:12.018858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c755d8-c14d-4b07-87d9-4f9ea2e5bb61 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:12.019291 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c755d8-c14d-4b07-87d9-4f9ea2e5bb61 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:12.119959 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:12.135980 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c755d8-c14d-4b07-87d9-4f9ea2e5bb61 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:12.137469 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c755d8-c14d-4b07-87d9-4f9ea2e5bb61 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:12.143078 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc7ff11-c31c-4f25-85ce-96e3a67c75f8 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:12.183649 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b56ea8-5adf-4b8f-b0fa-65f68eb567ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:12.184378 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b56ea8-5adf-4b8f-b0fa-65f68eb567ea None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:12.184740 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b56ea8-5adf-4b8f-b0fa-65f68eb567ea None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:12.185199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b56ea8-5adf-4b8f-b0fa-65f68eb567ea None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:12.185970 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b56ea8-5adf-4b8f-b0fa-65f68eb567ea None admin] RBAC: Adding query filter params (name=admin) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:12.186375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b56ea8-5adf-4b8f-b0fa-65f68eb567ea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:12.187931 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b56ea8-5adf-4b8f-b0fa-65f68eb567ea None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:12.198026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 89/178] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:12.199460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b56ea8-5adf-4b8f-b0fa-65f68eb567ea None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:12.266619 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 90/179] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:42:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 501 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:42:12.268650 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e433bb1-a905-488e-8b3f-9f89119f6f5e None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:12.269136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e433bb1-a905-488e-8b3f-9f89119f6f5e None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:12.269605 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e433bb1-a905-488e-8b3f-9f89119f6f5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:12.279041 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:12.292144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-6e433bb1-a905-488e-8b3f-9f89119f6f5e None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:12.293631 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-6e433bb1-a905-488e-8b3f-9f89119f6f5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:12.362875 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b56ea8-5adf-4b8f-b0fa-65f68eb567ea None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:12.386491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 91/180] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:12.386491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e433bb1-a905-488e-8b3f-9f89119f6f5e None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:12.425720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5576ffd-b2f2-4564-9c6e-fbe1c0ce98e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:12.426779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5576ffd-b2f2-4564-9c6e-fbe1c0ce98e3 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:12.427161 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5576ffd-b2f2-4564-9c6e-fbe1c0ce98e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:12.427548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5576ffd-b2f2-4564-9c6e-fbe1c0ce98e3 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:12.429026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5576ffd-b2f2-4564-9c6e-fbe1c0ce98e3 None admin] RBAC: Adding query filter params (name=member) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:12.429428 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5576ffd-b2f2-4564-9c6e-fbe1c0ce98e3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:12.432026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5576ffd-b2f2-4564-9c6e-fbe1c0ce98e3 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:12.447088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 90/181] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:12 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:12.448099 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5576ffd-b2f2-4564-9c6e-fbe1c0ce98e3 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:12.466844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:12.563433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e433bb1-a905-488e-8b3f-9f89119f6f5e None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:12.610025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ce389e-4ec2-4b26-8463-6d491eeb2e55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:12.610871 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ce389e-4ec2-4b26-8463-6d491eeb2e55 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:12.611265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ce389e-4ec2-4b26-8463-6d491eeb2e55 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:12.611647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ce389e-4ec2-4b26-8463-6d491eeb2e55 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:12.615916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:12.628045 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ce389e-4ec2-4b26-8463-6d491eeb2e55 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:12.634561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-d9ce389e-4ec2-4b26-8463-6d491eeb2e55 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:42:12.634561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:12.634561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:12.634561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:12.634561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:12.634561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:12.634561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:12.634561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:12.634561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:12.634561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:12.634561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:12.634561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:12.634561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:12.634561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:12.634561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:12.634561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:12.636539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:12.636539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:12.636539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:12.636539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:12.636539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:12.636539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:12.636539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:12.636539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:12.636539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:12.636539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:12.636539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:12.636539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:12.636539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:12.636539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:12.636539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:12.636539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:12.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:12.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:12.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:12.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:12.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:12.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:12.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:12.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:12.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:12.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:12.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:42:12.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:42:12.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 92/182] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 18 00:42:12 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:12.644490 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86335d48-cf05-456f-b22b-a1de754be241 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:12.644918 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86335d48-cf05-456f-b22b-a1de754be241 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:12.645331 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86335d48-cf05-456f-b22b-a1de754be241 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:12.648022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 93/183] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:12 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:12.648508 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86335d48-cf05-456f-b22b-a1de754be241 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:12.662484 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5576ffd-b2f2-4564-9c6e-fbe1c0ce98e3 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:12.707356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954f9d8d-fe9f-4466-b85c-154191f1978d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:12.708056 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954f9d8d-fe9f-4466-b85c-154191f1978d None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:12.708397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954f9d8d-fe9f-4466-b85c-154191f1978d None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:12.708744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954f9d8d-fe9f-4466-b85c-154191f1978d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:12.709644 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954f9d8d-fe9f-4466-b85c-154191f1978d None admin] RBAC: Adding query filter params (role.id=18f6cee478e54567b70d5c7557fe694e, scope.project.id=c3a25b7f689e4e21b2f61ed886a2a2d2, user.id=796c8ea379cf4e7c9570bc2fdba53211) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:12.709993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954f9d8d-fe9f-4466-b85c-154191f1978d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:12.711694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954f9d8d-fe9f-4466-b85c-154191f1978d None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:12.720408 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 91/184] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:42:12 2022] GET /identity/v3/role_assignments?user.id=796c8ea379cf4e7c9570bc2fdba53211&scope.project.id=c3a25b7f689e4e21b2f61ed886a2a2d2&role.id=18f6cee478e54567b70d5c7557fe694e => generated 608 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:12.721520 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954f9d8d-fe9f-4466-b85c-154191f1978d None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:12.825304 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86335d48-cf05-456f-b22b-a1de754be241 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:12.845943 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:12.882279 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01ac20d-e3dd-4cc1-a05c-4e0357b0450d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:12.883197 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01ac20d-e3dd-4cc1-a05c-4e0357b0450d None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:12.883607 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01ac20d-e3dd-4cc1-a05c-4e0357b0450d None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:12.884012 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01ac20d-e3dd-4cc1-a05c-4e0357b0450d None admin] PATH_INFO: `/v3/users/ba85fdf216fc4cd19bda312f5da53183` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:12.903681 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01ac20d-e3dd-4cc1-a05c-4e0357b0450d None admin] RBAC: Authorizing `identity:get_user(user_id=ba85fdf216fc4cd19bda312f5da53183)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:12.905629 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01ac20d-e3dd-4cc1-a05c-4e0357b0450d None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:12.907838 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 94/185] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:12 2022] GET /identity/v3/users/ba85fdf216fc4cd19bda312f5da53183 => generated 283 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:12.909249 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01ac20d-e3dd-4cc1-a05c-4e0357b0450d None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:12.910809 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954f9d8d-fe9f-4466-b85c-154191f1978d None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:12.975067 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff167725-45a7-4d7c-adc0-853748c64595 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:12.975939 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff167725-45a7-4d7c-adc0-853748c64595 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:12.976348 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff167725-45a7-4d7c-adc0-853748c64595 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:12.976754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff167725-45a7-4d7c-adc0-853748c64595 None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:12.978799 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff167725-45a7-4d7c-adc0-853748c64595 None admin] RBAC: Authorizing `identity:get_project(project_id=4c46ff2118f948a085f639b37068aa57)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:12.980893 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff167725-45a7-4d7c-adc0-853748c64595 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:12.984090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 92/186] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:12 2022] GET /identity/v3/projects/4c46ff2118f948a085f639b37068aa57 => generated 298 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:12.985140 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff167725-45a7-4d7c-adc0-853748c64595 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:12.991904 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:13.076758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01ac20d-e3dd-4cc1-a05c-4e0357b0450d None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:13.121379 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438e0a6b-c24c-48d0-9e2b-96108bfd4ffb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:13.122028 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438e0a6b-c24c-48d0-9e2b-96108bfd4ffb None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:13.122874 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438e0a6b-c24c-48d0-9e2b-96108bfd4ffb None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:13.123150 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438e0a6b-c24c-48d0-9e2b-96108bfd4ffb None admin] PATH_INFO: `/v3/users/796c8ea379cf4e7c9570bc2fdba53211` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:13.124363 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438e0a6b-c24c-48d0-9e2b-96108bfd4ffb None admin] RBAC: Authorizing `identity:get_user(user_id=796c8ea379cf4e7c9570bc2fdba53211)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:13.125841 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438e0a6b-c24c-48d0-9e2b-96108bfd4ffb None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:13.127544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 95/187] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:12 2022] GET /identity/v3/users/796c8ea379cf4e7c9570bc2fdba53211 => generated 275 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:13.128753 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777bc9be-1a72-4e3c-be53-25f576157cb1 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:13.129031 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777bc9be-1a72-4e3c-be53-25f576157cb1 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:13.129307 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777bc9be-1a72-4e3c-be53-25f576157cb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:13.148460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff167725-45a7-4d7c-adc0-853748c64595 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:13.148922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-777bc9be-1a72-4e3c-be53-25f576157cb1 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:13.150156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-777bc9be-1a72-4e3c-be53-25f576157cb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:13.187681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fd65b0-7a7e-4ed1-87f1-a187c5e6d8d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:13.188306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fd65b0-7a7e-4ed1-87f1-a187c5e6d8d7 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:13.188586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fd65b0-7a7e-4ed1-87f1-a187c5e6d8d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:13.188855 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fd65b0-7a7e-4ed1-87f1-a187c5e6d8d7 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:13.189538 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fd65b0-7a7e-4ed1-87f1-a187c5e6d8d7 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:13.189845 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fd65b0-7a7e-4ed1-87f1-a187c5e6d8d7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:13.191287 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fd65b0-7a7e-4ed1-87f1-a187c5e6d8d7 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:13.197101 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:13.202165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 93/188] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:42:12 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:13.204121 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fd65b0-7a7e-4ed1-87f1-a187c5e6d8d7 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:13.236731 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 96/189] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:13.243758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777bc9be-1a72-4e3c-be53-25f576157cb1 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:13.333733 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:13.384221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fd65b0-7a7e-4ed1-87f1-a187c5e6d8d7 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:13.421224 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777bc9be-1a72-4e3c-be53-25f576157cb1 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:13.426452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac7dea2-dec4-476f-b469-3d978a4c0550 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:13.427304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac7dea2-dec4-476f-b469-3d978a4c0550 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:13.427434 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac7dea2-dec4-476f-b469-3d978a4c0550 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:13.427898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac7dea2-dec4-476f-b469-3d978a4c0550 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:13.428887 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac7dea2-dec4-476f-b469-3d978a4c0550 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:13.430717 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac7dea2-dec4-476f-b469-3d978a4c0550 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:13.432470 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 94/190] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:42:13 2022] GET /identity/v3/domains/default => generated 199 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:13.433280 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac7dea2-dec4-476f-b469-3d978a4c0550 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:13.493716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80898d3d-ff3d-44d7-8197-e21a97ada6ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:13.495172 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80898d3d-ff3d-44d7-8197-e21a97ada6ce None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:13.495505 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80898d3d-ff3d-44d7-8197-e21a97ada6ce None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:13.495745 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80898d3d-ff3d-44d7-8197-e21a97ada6ce None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:13.552429 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:13.598450 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac7dea2-dec4-476f-b469-3d978a4c0550 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:13.608070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80898d3d-ff3d-44d7-8197-e21a97ada6ce None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:13.609566 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80898d3d-ff3d-44d7-8197-e21a97ada6ce None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:13.635059 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0610b4-fb19-4e60-af6f-166b79eb5181 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:13.635686 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0610b4-fb19-4e60-af6f-166b79eb5181 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:13.635965 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0610b4-fb19-4e60-af6f-166b79eb5181 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:13.636236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0610b4-fb19-4e60-af6f-166b79eb5181 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:13.637005 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0610b4-fb19-4e60-af6f-166b79eb5181 None admin] RBAC: Adding query filter params (role.id=cf934cf252ce42d5a0445d3e156348d1, scope.project.id=4c46ff2118f948a085f639b37068aa57, user.id=796c8ea379cf4e7c9570bc2fdba53211) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:13.637311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0610b4-fb19-4e60-af6f-166b79eb5181 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:13.639376 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0610b4-fb19-4e60-af6f-166b79eb5181 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:13.645730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 95/191] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:42:13 2022] GET /identity/v3/role_assignments?user.id=796c8ea379cf4e7c9570bc2fdba53211&scope.project.id=4c46ff2118f948a085f639b37068aa57&role.id=cf934cf252ce42d5a0445d3e156348d1 => generated 608 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:13.646665 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0610b4-fb19-4e60-af6f-166b79eb5181 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:13.667241 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:13.745223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 97/192] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:42:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 508 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:42:13.746542 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80898d3d-ff3d-44d7-8197-e21a97ada6ce None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:13.817858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0610b4-fb19-4e60-af6f-166b79eb5181 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:13.861684 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:13.869547 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d381aae6-019c-446f-a6ac-73763602b03f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:13.870327 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d381aae6-019c-446f-a6ac-73763602b03f None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:13.871183 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d381aae6-019c-446f-a6ac-73763602b03f None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:13.871364 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d381aae6-019c-446f-a6ac-73763602b03f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:13.885552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80898d3d-ff3d-44d7-8197-e21a97ada6ce None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:13.934402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e1ecf4-885c-4aca-a097-ad0e007570c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:13.934402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e1ecf4-885c-4aca-a097-ad0e007570c5 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:13.934402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e1ecf4-885c-4aca-a097-ad0e007570c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:13.935466 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e1ecf4-885c-4aca-a097-ad0e007570c5 None admin] PATH_INFO: `/v3/users/ba85fdf216fc4cd19bda312f5da53183` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:13.936634 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e1ecf4-885c-4aca-a097-ad0e007570c5 None admin] RBAC: Authorizing `identity:get_user(user_id=ba85fdf216fc4cd19bda312f5da53183)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:13.938175 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e1ecf4-885c-4aca-a097-ad0e007570c5 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:13.939914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 98/193] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:13 2022] GET /identity/v3/users/ba85fdf216fc4cd19bda312f5da53183 => generated 283 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:13.940842 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e1ecf4-885c-4aca-a097-ad0e007570c5 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:13.962258 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:13.976461 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d381aae6-019c-446f-a6ac-73763602b03f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:13.978038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d381aae6-019c-446f-a6ac-73763602b03f None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:14.054398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e1ecf4-885c-4aca-a097-ad0e007570c5 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:14.096126 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 96/194] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:42:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 450 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:42:14.097090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d381aae6-019c-446f-a6ac-73763602b03f None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:14.099285 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e34138-047d-4680-bb3f-1fd994e558a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:14.100227 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e34138-047d-4680-bb3f-1fd994e558a0 None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:14.100788 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e34138-047d-4680-bb3f-1fd994e558a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:14.101226 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e34138-047d-4680-bb3f-1fd994e558a0 None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:14.102256 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e34138-047d-4680-bb3f-1fd994e558a0 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:14.103820 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e34138-047d-4680-bb3f-1fd994e558a0 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:14.135461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 99/195] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 18 00:42:13 2022] POST /identity/v3/users => generated 297 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:42:14.135638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:14.141925 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e34138-047d-4680-bb3f-1fd994e558a0 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:14.239812 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d381aae6-019c-446f-a6ac-73763602b03f None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:14.264058 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:14.286445 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bdb1ac-56cd-45f5-9bcf-5d07e1e8933c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:14.287292 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bdb1ac-56cd-45f5-9bcf-5d07e1e8933c None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:14.287766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bdb1ac-56cd-45f5-9bcf-5d07e1e8933c None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:14.288142 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bdb1ac-56cd-45f5-9bcf-5d07e1e8933c None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:14.300508 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e34138-047d-4680-bb3f-1fd994e558a0 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:14.302115 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bdb1ac-56cd-45f5-9bcf-5d07e1e8933c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:14.307332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-f6bdb1ac-56cd-45f5-9bcf-5d07e1e8933c None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 18 00:42:14.307332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:14.307332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:14.307332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:14.307332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:14.307332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:14.307332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:14.307332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:14.307332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:14.307332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:14.307332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:14.307332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:14.307332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:14.307332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:14.307332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:14.307332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:14.309029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:14.309029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:14.309029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:14.309029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:14.309029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:14.309029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:14.309029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:14.309029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:14.309029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:14.309029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:14.309029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:14.309029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:14.309029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:14.309029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:14.309029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:14.309029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:14.310636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:14.310636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:14.310636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:14.310636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:14.310636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:14.310636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:14.310636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:14.310636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:14.310636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:14.310636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:14.310636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 18 00:42:14.310636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:42:14.310636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 97/196] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:14 2022] GET /identity/v3/roles/reader => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:14.310636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bdb1ac-56cd-45f5-9bcf-5d07e1e8933c None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:14.353120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f2edee-e0e1-49c0-a395-143be9f33629 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:14.353766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f2edee-e0e1-49c0-a395-143be9f33629 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:14.354050 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f2edee-e0e1-49c0-a395-143be9f33629 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:14.354323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f2edee-e0e1-49c0-a395-143be9f33629 None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:14.355599 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f2edee-e0e1-49c0-a395-143be9f33629 None admin] RBAC: Authorizing `identity:get_project(project_id=8085b2496c6847aa83e809bde1f3e986)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:14.358545 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f2edee-e0e1-49c0-a395-143be9f33629 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:14.360406 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 100/197] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:14 2022] GET /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986 => generated 302 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:14.360406 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f2edee-e0e1-49c0-a395-143be9f33629 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:14.454014 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:14.469667 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bdb1ac-56cd-45f5-9bcf-5d07e1e8933c None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:14.510830 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f2edee-e0e1-49c0-a395-143be9f33629 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:14.537656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e841a032-edf7-4ee7-9631-14e6ef68f3b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:14.538292 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e841a032-edf7-4ee7-9631-14e6ef68f3b3 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:14.538646 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e841a032-edf7-4ee7-9631-14e6ef68f3b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:14.538919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e841a032-edf7-4ee7-9631-14e6ef68f3b3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:14.573642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9424b35-d9d8-4f91-bb4a-26b400ed58cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:14.574383 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9424b35-d9d8-4f91-bb4a-26b400ed58cd None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:14.574922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9424b35-d9d8-4f91-bb4a-26b400ed58cd None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:14.575885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9424b35-d9d8-4f91-bb4a-26b400ed58cd None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:14.577841 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9424b35-d9d8-4f91-bb4a-26b400ed58cd None admin] RBAC: Authorizing `identity:get_project(project_id=4c46ff2118f948a085f639b37068aa57)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:14.581421 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9424b35-d9d8-4f91-bb4a-26b400ed58cd None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:14.584316 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 101/198] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:14 2022] GET /identity/v3/projects/4c46ff2118f948a085f639b37068aa57 => generated 298 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:14.585386 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9424b35-d9d8-4f91-bb4a-26b400ed58cd None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:14.609825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:14.627164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e841a032-edf7-4ee7-9631-14e6ef68f3b3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:14.630118 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e841a032-edf7-4ee7-9631-14e6ef68f3b3 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:14.761941 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:14.772347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9424b35-d9d8-4f91-bb4a-26b400ed58cd None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:14.774178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 98/199] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:42:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 465 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:42:14.775108 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e841a032-edf7-4ee7-9631-14e6ef68f3b3 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:14.843518 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c742fad-303f-45fe-ba32-3f632e735de1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:14.844847 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c742fad-303f-45fe-ba32-3f632e735de1 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:14.845534 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c742fad-303f-45fe-ba32-3f632e735de1 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:14.846318 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c742fad-303f-45fe-ba32-3f632e735de1 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:14.847855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c742fad-303f-45fe-ba32-3f632e735de1 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:14.848600 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c742fad-303f-45fe-ba32-3f632e735de1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:14.851672 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c742fad-303f-45fe-ba32-3f632e735de1 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:14.869253 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 102/200] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:14 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:14.879794 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c742fad-303f-45fe-ba32-3f632e735de1 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:14.908293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e841a032-edf7-4ee7-9631-14e6ef68f3b3 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:14.951011 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c33829-81c4-4181-a1f1-14ec07256d6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:14.951728 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c33829-81c4-4181-a1f1-14ec07256d6a None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:14.952545 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c33829-81c4-4181-a1f1-14ec07256d6a None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:14.952673 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c33829-81c4-4181-a1f1-14ec07256d6a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:14.953458 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c33829-81c4-4181-a1f1-14ec07256d6a None admin] RBAC: Adding query filter params (role.id=1e04c9fea65244d28e1e59fc4c64f32f, scope.project.id=8085b2496c6847aa83e809bde1f3e986, user.id=ba85fdf216fc4cd19bda312f5da53183) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:14.953773 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c33829-81c4-4181-a1f1-14ec07256d6a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:14.955306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c33829-81c4-4181-a1f1-14ec07256d6a None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:14.962200 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 99/201] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:42:14 2022] GET /identity/v3/role_assignments?user.id=ba85fdf216fc4cd19bda312f5da53183&scope.project.id=8085b2496c6847aa83e809bde1f3e986&role.id=1e04c9fea65244d28e1e59fc4c64f32f => generated 259 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:14.963236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c33829-81c4-4181-a1f1-14ec07256d6a None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:15.014939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:15.079524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c742fad-303f-45fe-ba32-3f632e735de1 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:15.095014 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:15.155059 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881921d5-63f5-42fb-aaa0-a888520dc2ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:15.155909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881921d5-63f5-42fb-aaa0-a888520dc2ba None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:15.156290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881921d5-63f5-42fb-aaa0-a888520dc2ba None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:15.156746 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881921d5-63f5-42fb-aaa0-a888520dc2ba None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:15.157617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881921d5-63f5-42fb-aaa0-a888520dc2ba None admin] RBAC: Adding query filter params (role.id=18f6cee478e54567b70d5c7557fe694e, scope.project.id=4c46ff2118f948a085f639b37068aa57, user.id=796c8ea379cf4e7c9570bc2fdba53211) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:15.158047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881921d5-63f5-42fb-aaa0-a888520dc2ba None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:15.159864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881921d5-63f5-42fb-aaa0-a888520dc2ba None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:15.173846 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 103/202] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:42:14 2022] GET /identity/v3/role_assignments?user.id=796c8ea379cf4e7c9570bc2fdba53211&scope.project.id=4c46ff2118f948a085f639b37068aa57&role.id=18f6cee478e54567b70d5c7557fe694e => generated 608 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:15.175060 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881921d5-63f5-42fb-aaa0-a888520dc2ba None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:15.204871 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c33829-81c4-4181-a1f1-14ec07256d6a None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:15.322899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be95c479-e66f-4783-aff8-b095d13fe669 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:15.323547 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be95c479-e66f-4783-aff8-b095d13fe669 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:15.323822 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be95c479-e66f-4783-aff8-b095d13fe669 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:15.324082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be95c479-e66f-4783-aff8-b095d13fe669 None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:15.326715 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be95c479-e66f-4783-aff8-b095d13fe669 None admin] RBAC: Authorizing `identity:get_project(project_id=8085b2496c6847aa83e809bde1f3e986)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:15.328429 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be95c479-e66f-4783-aff8-b095d13fe669 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:15.330328 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 100/203] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:14 2022] GET /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986 => generated 302 bytes in 368 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:15.338799 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be95c479-e66f-4783-aff8-b095d13fe669 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:15.432515 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881921d5-63f5-42fb-aaa0-a888520dc2ba None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:15.507691 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d586de-fda4-473d-ae1c-c23bb07bc1c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:15.509337 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d586de-fda4-473d-ae1c-c23bb07bc1c7 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:15.510352 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d586de-fda4-473d-ae1c-c23bb07bc1c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:15.515443 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d586de-fda4-473d-ae1c-c23bb07bc1c7 None admin] PATH_INFO: `/v3/users/42fc7c5de9eb4a7cb41e4c636c8148e3` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:15.518436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d586de-fda4-473d-ae1c-c23bb07bc1c7 None admin] RBAC: Authorizing `identity:get_user(user_id=42fc7c5de9eb4a7cb41e4c636c8148e3)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:15.525361 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d586de-fda4-473d-ae1c-c23bb07bc1c7 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:15.529998 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 104/204] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:15 2022] GET /identity/v3/users/42fc7c5de9eb4a7cb41e4c636c8148e3 => generated 289 bytes in 356 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:15.539358 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d586de-fda4-473d-ae1c-c23bb07bc1c7 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:15.547390 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:15.631986 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be95c479-e66f-4783-aff8-b095d13fe669 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:15.683368 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:15.698167 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42cbfee-19ec-461f-8b24-e281c73bb618 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:15.699236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42cbfee-19ec-461f-8b24-e281c73bb618 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:15.699236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42cbfee-19ec-461f-8b24-e281c73bb618 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:15.699531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42cbfee-19ec-461f-8b24-e281c73bb618 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:15.700102 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42cbfee-19ec-461f-8b24-e281c73bb618 None admin] RBAC: Adding query filter params (role.id=cf934cf252ce42d5a0445d3e156348d1, scope.project.id=8085b2496c6847aa83e809bde1f3e986, user.id=ba85fdf216fc4cd19bda312f5da53183) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:15.700496 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42cbfee-19ec-461f-8b24-e281c73bb618 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:15.702074 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42cbfee-19ec-461f-8b24-e281c73bb618 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:15.708662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 101/205] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:42:15 2022] GET /identity/v3/role_assignments?user.id=ba85fdf216fc4cd19bda312f5da53183&scope.project.id=8085b2496c6847aa83e809bde1f3e986&role.id=cf934cf252ce42d5a0445d3e156348d1 => generated 259 bytes in 370 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:15.861596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d586de-fda4-473d-ae1c-c23bb07bc1c7 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:15.936483 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804ea632-cd76-4b98-a5dc-a9f3f57dbcdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:15.937955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804ea632-cd76-4b98-a5dc-a9f3f57dbcdf None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:15.938688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804ea632-cd76-4b98-a5dc-a9f3f57dbcdf None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:15.939403 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804ea632-cd76-4b98-a5dc-a9f3f57dbcdf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:16.088126 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804ea632-cd76-4b98-a5dc-a9f3f57dbcdf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:16.091728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804ea632-cd76-4b98-a5dc-a9f3f57dbcdf None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:16.097546 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:16.301002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 105/206] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:42:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 763 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:42:16.308237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42cbfee-19ec-461f-8b24-e281c73bb618 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:16.454007 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42cbfee-19ec-461f-8b24-e281c73bb618 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:16.515058 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2510de4-9e74-4e89-a606-6db9a4442940 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:16.516144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2510de4-9e74-4e89-a606-6db9a4442940 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:16.516635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2510de4-9e74-4e89-a606-6db9a4442940 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:16.517109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2510de4-9e74-4e89-a606-6db9a4442940 None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:16.519126 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2510de4-9e74-4e89-a606-6db9a4442940 None admin] RBAC: Authorizing `identity:get_project(project_id=4c46ff2118f948a085f639b37068aa57)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:16.521775 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2510de4-9e74-4e89-a606-6db9a4442940 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:16.524901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 102/207] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:16 2022] GET /identity/v3/projects/4c46ff2118f948a085f639b37068aa57 => generated 298 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:16.531852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804ea632-cd76-4b98-a5dc-a9f3f57dbcdf None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:16.749869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804ea632-cd76-4b98-a5dc-a9f3f57dbcdf None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:16.760586 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:16.844429 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa36bc60-a118-41e4-8450-ec6cc22e123e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:16.845477 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa36bc60-a118-41e4-8450-ec6cc22e123e None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:16.845958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa36bc60-a118-41e4-8450-ec6cc22e123e None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:16.846446 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa36bc60-a118-41e4-8450-ec6cc22e123e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:16.847868 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa36bc60-a118-41e4-8450-ec6cc22e123e None admin] RBAC: Adding query filter params (role.id=a928285094b34996a609520c255b4c5d, scope.project.id=4c46ff2118f948a085f639b37068aa57, user.id=42fc7c5de9eb4a7cb41e4c636c8148e3) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:16.848412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa36bc60-a118-41e4-8450-ec6cc22e123e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:16.851209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa36bc60-a118-41e4-8450-ec6cc22e123e None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:16.867177 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 106/208] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:42:16 2022] GET /identity/v3/role_assignments?user.id=42fc7c5de9eb4a7cb41e4c636c8148e3&scope.project.id=4c46ff2118f948a085f639b37068aa57&role.id=a928285094b34996a609520c255b4c5d => generated 608 bytes in 336 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:17.184537 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fd06e6-b9ab-4729-bbad-f47739ab7a30 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:17.185590 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fd06e6-b9ab-4729-bbad-f47739ab7a30 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:17.186035 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fd06e6-b9ab-4729-bbad-f47739ab7a30 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:17.189018 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 103/209] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:17 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:17.198340 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c02d8d-2b4b-410d-b0ac-2b88ac176269 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:17.198852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c02d8d-2b4b-410d-b0ac-2b88ac176269 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:17.199352 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c02d8d-2b4b-410d-b0ac-2b88ac176269 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:17.240332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-c4c02d8d-2b4b-410d-b0ac-2b88ac176269 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:17.242391 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c4c02d8d-2b4b-410d-b0ac-2b88ac176269 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:17.377330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 107/210] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:17.963701 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a85336-9974-4f74-962a-be4b0409d132 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:17.964207 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a85336-9974-4f74-962a-be4b0409d132 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:17.964703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a85336-9974-4f74-962a-be4b0409d132 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:17.967338 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 104/211] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:17 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:17.972261 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c02d8d-2b4b-410d-b0ac-2b88ac176269 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:17.976583 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce049687-f299-43a0-a1fd-bac336faa760 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:17.977062 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce049687-f299-43a0-a1fd-bac336faa760 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:17.977510 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce049687-f299-43a0-a1fd-bac336faa760 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:18.013622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-ce049687-f299-43a0-a1fd-bac336faa760 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:18.015770 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ce049687-f299-43a0-a1fd-bac336faa760 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:18.119826 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:18.162999 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 105/212] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:42:18.184207 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d86c442-07e5-4f20-87fb-daee6b31c0b9 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:18.184831 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d86c442-07e5-4f20-87fb-daee6b31c0b9 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:18.185354 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d86c442-07e5-4f20-87fb-daee6b31c0b9 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:18.188321 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 106/213] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:18 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:18.194370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c02d8d-2b4b-410d-b0ac-2b88ac176269 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:18.195460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a940d61-96c6-49bf-b3a5-aa1725d1d942 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:18.195962 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a940d61-96c6-49bf-b3a5-aa1725d1d942 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:18.196426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a940d61-96c6-49bf-b3a5-aa1725d1d942 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:18.199655 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:18.232530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-3a940d61-96c6-49bf-b3a5-aa1725d1d942 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:18.235149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-3a940d61-96c6-49bf-b3a5-aa1725d1d942 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:18.245663 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788c5430-954e-42f6-a25c-493051ee9d7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:18.247560 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788c5430-954e-42f6-a25c-493051ee9d7a None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:18.248623 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788c5430-954e-42f6-a25c-493051ee9d7a None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:18.249404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788c5430-954e-42f6-a25c-493051ee9d7a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:18.271801 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788c5430-954e-42f6-a25c-493051ee9d7a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:18.283407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-788c5430-954e-42f6-a25c-493051ee9d7a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:18.283407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:18.283407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:18.283407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:18.283407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:18.283407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:18.283407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:18.283407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:18.283407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:18.283407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:18.283407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:18.283407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:18.283407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:18.283407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:18.283407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:18.283407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:18.285131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:18.285131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:18.285131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:18.285131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:18.285131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:18.285131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:18.285131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:18.285131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:18.285131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:18.285131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:18.285131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:18.285131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:18.285131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:18.285131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:18.285131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:18.285131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:18.286785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:18.286785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:18.286785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:18.286785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:18.286785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:18.286785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:18.286785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:18.286785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:18.286785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:18.286785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:18.286785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:18.286785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:42:18.291452 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 108/214] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:17 2022] GET /identity/v3/roles/member => generated 84 bytes in 320 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:18.302001 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788c5430-954e-42f6-a25c-493051ee9d7a None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:18.326604 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 107/215] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:18.429939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788c5430-954e-42f6-a25c-493051ee9d7a None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:18.471634 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b726a33-3680-4637-ab5d-ea20ad505879 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:18.472273 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b726a33-3680-4637-ab5d-ea20ad505879 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:18.472561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b726a33-3680-4637-ab5d-ea20ad505879 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:18.472837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b726a33-3680-4637-ab5d-ea20ad505879 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:18.473540 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b726a33-3680-4637-ab5d-ea20ad505879 None admin] RBAC: Adding query filter params (name=member) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:18.473846 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b726a33-3680-4637-ab5d-ea20ad505879 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:18.475335 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b726a33-3680-4637-ab5d-ea20ad505879 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:18.485210 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 109/216] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:18.491256 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a940d61-96c6-49bf-b3a5-aa1725d1d942 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:18.616161 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a940d61-96c6-49bf-b3a5-aa1725d1d942 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:18.634981 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:18.646870 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b726a33-3680-4637-ab5d-ea20ad505879 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:18.656852 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27ebeb1-151e-4f11-9304-57007718d63a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:18.657615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27ebeb1-151e-4f11-9304-57007718d63a None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:18.657895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27ebeb1-151e-4f11-9304-57007718d63a None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:18.658164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27ebeb1-151e-4f11-9304-57007718d63a None admin] PATH_INFO: `/v3/users/6e1b221d671c437cb5f87821a5d327ac` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:18.672774 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:18.676412 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27ebeb1-151e-4f11-9304-57007718d63a None admin] RBAC: Authorizing `identity:get_user(user_id=6e1b221d671c437cb5f87821a5d327ac)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:18.677947 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27ebeb1-151e-4f11-9304-57007718d63a None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:18.679757 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 108/217] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:18 2022] GET /identity/v3/users/6e1b221d671c437cb5f87821a5d327ac => generated 297 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:18.681671 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27ebeb1-151e-4f11-9304-57007718d63a None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:18.771845 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b726a33-3680-4637-ab5d-ea20ad505879 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:18.822161 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb678bd-28b1-444c-a702-8941811b15cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:18.822712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb678bd-28b1-444c-a702-8941811b15cc None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:18.822991 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb678bd-28b1-444c-a702-8941811b15cc None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:18.823261 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb678bd-28b1-444c-a702-8941811b15cc None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:18.832712 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27ebeb1-151e-4f11-9304-57007718d63a None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:18.837934 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb678bd-28b1-444c-a702-8941811b15cc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:18.844155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-cbb678bd-28b1-444c-a702-8941811b15cc None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:42:18.844155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:18.844155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:18.844155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:18.844155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:18.844155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:18.844155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:18.844155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:18.844155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:18.844155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:18.844155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:18.844155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:18.844155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:18.844155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:18.844155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:18.844155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:18.846051 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:18.846051 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:18.846051 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:18.846051 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:18.846051 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:18.846051 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:18.846051 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:18.846051 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:18.846051 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:18.846051 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:18.846051 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:18.846051 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:18.846051 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:18.846051 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:18.846051 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:18.846051 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:18.849458 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:18.849458 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:18.849458 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:18.849458 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:18.849458 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:18.849458 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:18.849458 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:18.849458 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:18.849458 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:18.849458 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:18.849458 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:42:18.849458 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:42:18.849458 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 110/218] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:42:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:18.849458 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb678bd-28b1-444c-a702-8941811b15cc None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:18.887060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592ab509-d5a4-47be-a5a6-7fe7ab485261 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:18.888225 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592ab509-d5a4-47be-a5a6-7fe7ab485261 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:18.888515 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592ab509-d5a4-47be-a5a6-7fe7ab485261 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:18.888811 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592ab509-d5a4-47be-a5a6-7fe7ab485261 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:18.890070 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592ab509-d5a4-47be-a5a6-7fe7ab485261 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:18.891583 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592ab509-d5a4-47be-a5a6-7fe7ab485261 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:18.893240 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 109/219] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:42:18 2022] GET /identity/v3/domains/default => generated 199 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:18.893240 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592ab509-d5a4-47be-a5a6-7fe7ab485261 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:18.953694 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:18.962589 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:18.977072 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb678bd-28b1-444c-a702-8941811b15cc None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:19.024366 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49a4e70-20de-4eae-b74c-917ee879be39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:19.024926 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49a4e70-20de-4eae-b74c-917ee879be39 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:19.025216 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49a4e70-20de-4eae-b74c-917ee879be39 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:19.025490 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49a4e70-20de-4eae-b74c-917ee879be39 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:19.037783 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592ab509-d5a4-47be-a5a6-7fe7ab485261 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:19.083639 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8158aa0b-e919-477d-9098-92f7f3608e60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:19.084664 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8158aa0b-e919-477d-9098-92f7f3608e60 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:19.085206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8158aa0b-e919-477d-9098-92f7f3608e60 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:19.085682 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8158aa0b-e919-477d-9098-92f7f3608e60 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:19.086685 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8158aa0b-e919-477d-9098-92f7f3608e60 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:19.087176 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8158aa0b-e919-477d-9098-92f7f3608e60 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:19.088766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8158aa0b-e919-477d-9098-92f7f3608e60 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:19.105643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 110/220] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:19.106918 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8158aa0b-e919-477d-9098-92f7f3608e60 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:19.116803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49a4e70-20de-4eae-b74c-917ee879be39 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:19.118358 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49a4e70-20de-4eae-b74c-917ee879be39 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:19.239780 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:19.244560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8158aa0b-e919-477d-9098-92f7f3608e60 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:19.248687 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:19.268572 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 111/221] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:42:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:42:19.271772 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49a4e70-20de-4eae-b74c-917ee879be39 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:19.298257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f58613-d223-490c-b973-896e1a5d4d49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:19.299116 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f58613-d223-490c-b973-896e1a5d4d49 None admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:19.299813 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f58613-d223-490c-b973-896e1a5d4d49 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:19.300240 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f58613-d223-490c-b973-896e1a5d4d49 None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:19.301249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f58613-d223-490c-b973-896e1a5d4d49 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:19.303016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f58613-d223-490c-b973-896e1a5d4d49 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:19.328076 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 111/222] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 18 00:42:19 2022] POST /identity/v3/users => generated 297 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:42:19.328924 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f58613-d223-490c-b973-896e1a5d4d49 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:19.425094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49a4e70-20de-4eae-b74c-917ee879be39 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:19.461076 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f58613-d223-490c-b973-896e1a5d4d49 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:19.472353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac7da60-67ff-451c-b62a-38e10a4ba0bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:19.473105 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac7da60-67ff-451c-b62a-38e10a4ba0bc None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:19.473381 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac7da60-67ff-451c-b62a-38e10a4ba0bc None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:19.473646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac7da60-67ff-451c-b62a-38e10a4ba0bc None admin] PATH_INFO: `/v3/users/ba85fdf216fc4cd19bda312f5da53183` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:19.475153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac7da60-67ff-451c-b62a-38e10a4ba0bc None admin] RBAC: Authorizing `identity:get_user(user_id=ba85fdf216fc4cd19bda312f5da53183)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:19.476717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac7da60-67ff-451c-b62a-38e10a4ba0bc None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:19.478803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 112/223] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:19 2022] GET /identity/v3/users/ba85fdf216fc4cd19bda312f5da53183 => generated 283 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:19.485500 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3736c13-521e-4621-9e2f-66d18946cbe4 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:19.486052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3736c13-521e-4621-9e2f-66d18946cbe4 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:19.486351 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3736c13-521e-4621-9e2f-66d18946cbe4 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:19.488766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 113/224] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:19 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:19.490164 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3736c13-521e-4621-9e2f-66d18946cbe4 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:19.531187 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffc4b8e-0d9c-4ab2-acf6-38c08ea37f8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:19.532173 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffc4b8e-0d9c-4ab2-acf6-38c08ea37f8b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:19.532618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffc4b8e-0d9c-4ab2-acf6-38c08ea37f8b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:19.533070 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffc4b8e-0d9c-4ab2-acf6-38c08ea37f8b None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:19.535007 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffc4b8e-0d9c-4ab2-acf6-38c08ea37f8b None admin] RBAC: Authorizing `identity:get_project(project_id=8085b2496c6847aa83e809bde1f3e986)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:19.537275 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffc4b8e-0d9c-4ab2-acf6-38c08ea37f8b None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:19.545527 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 112/225] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:19 2022] GET /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986 => generated 302 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:19.545527 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa651efd-b54d-4813-9156-ca2d85b37920 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:19.545527 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa651efd-b54d-4813-9156-ca2d85b37920 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:19.545527 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa651efd-b54d-4813-9156-ca2d85b37920 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:19.581076 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-fa651efd-b54d-4813-9156-ca2d85b37920 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:19.583052 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-fa651efd-b54d-4813-9156-ca2d85b37920 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:19.589457 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:19.623330 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:19.676986 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3736c13-521e-4621-9e2f-66d18946cbe4 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:19.698817 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 113/226] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:19.701193 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa651efd-b54d-4813-9156-ca2d85b37920 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:19.751572 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e4fbde-813b-4117-baa8-cf0c7087ddba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:19.752608 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e4fbde-813b-4117-baa8-cf0c7087ddba None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:19.752895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e4fbde-813b-4117-baa8-cf0c7087ddba None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:19.753166 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e4fbde-813b-4117-baa8-cf0c7087ddba None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:19.755684 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e4fbde-813b-4117-baa8-cf0c7087ddba None admin] RBAC: Authorizing `identity:get_project(project_id=8085b2496c6847aa83e809bde1f3e986)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:19.756622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e4fbde-813b-4117-baa8-cf0c7087ddba None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:19.758426 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 114/227] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:19 2022] GET /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986 => generated 302 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:19.765459 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e4fbde-813b-4117-baa8-cf0c7087ddba None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:19.865270 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa651efd-b54d-4813-9156-ca2d85b37920 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:19.949343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97b7c77-4f75-4651-ba2d-ca6db28f99e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:19.953696 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97b7c77-4f75-4651-ba2d-ca6db28f99e6 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:19.954794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97b7c77-4f75-4651-ba2d-ca6db28f99e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:19.955576 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97b7c77-4f75-4651-ba2d-ca6db28f99e6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:19.958130 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97b7c77-4f75-4651-ba2d-ca6db28f99e6 None admin] RBAC: Adding query filter params (role.id=18f6cee478e54567b70d5c7557fe694e, scope.project.id=8085b2496c6847aa83e809bde1f3e986, user.id=6e1b221d671c437cb5f87821a5d327ac) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:19.959105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97b7c77-4f75-4651-ba2d-ca6db28f99e6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:19.963242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97b7c77-4f75-4651-ba2d-ca6db28f99e6 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:19.980318 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 114/228] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:42:19 2022] GET /identity/v3/role_assignments?user.id=6e1b221d671c437cb5f87821a5d327ac&scope.project.id=8085b2496c6847aa83e809bde1f3e986&role.id=18f6cee478e54567b70d5c7557fe694e => generated 259 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:20.013053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e4fbde-813b-4117-baa8-cf0c7087ddba None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:20.049627 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:20.087151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75842759-4008-45c4-9f11-0a78ed90a5d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:20.088401 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75842759-4008-45c4-9f11-0a78ed90a5d1 None admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:20.088944 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75842759-4008-45c4-9f11-0a78ed90a5d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:20.089426 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75842759-4008-45c4-9f11-0a78ed90a5d1 None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986/users/ba85fdf216fc4cd19bda312f5da53183/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:20.093829 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75842759-4008-45c4-9f11-0a78ed90a5d1 None admin] RBAC: Authorizing `identity:create_grant(project_id=8085b2496c6847aa83e809bde1f3e986, user_id=ba85fdf216fc4cd19bda312f5da53183, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:20.096697 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75842759-4008-45c4-9f11-0a78ed90a5d1 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:20.124159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 115/229] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:42:19 2022] PUT /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986/users/ba85fdf216fc4cd19bda312f5da53183/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 359 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:42:20.174153 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97b7c77-4f75-4651-ba2d-ca6db28f99e6 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:20.243926 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:20.419582 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97b7c77-4f75-4651-ba2d-ca6db28f99e6 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:20.491405 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b2ddef-ae9b-402a-8ae3-fa059ef815f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:20.492615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b2ddef-ae9b-402a-8ae3-fa059ef815f0 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:20.493097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b2ddef-ae9b-402a-8ae3-fa059ef815f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:20.493565 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b2ddef-ae9b-402a-8ae3-fa059ef815f0 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:20.519860 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b2ddef-ae9b-402a-8ae3-fa059ef815f0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:20.530290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-72b2ddef-ae9b-402a-8ae3-fa059ef815f0 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:42:20.530290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:20.530290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:20.530290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:20.530290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:20.530290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:20.530290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:20.530290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:20.530290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:20.530290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:20.530290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:20.530290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:20.530290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:20.530290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:20.530290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:20.530290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:20.531281 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:20.531281 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:20.531281 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:20.531281 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:20.531281 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:20.531281 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:20.531281 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:20.531281 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:20.531281 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:20.531281 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:20.531281 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:20.531281 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:20.531281 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:20.531281 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:20.531281 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:20.531281 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:20.532080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:20.532080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:20.532080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:20.532080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:20.532080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:20.532080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:20.532080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:20.532080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:20.532080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:20.532080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:20.532080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:42:20.532080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:42:20.538920 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 115/230] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 18 00:42:20 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 366 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:20.544573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75842759-4008-45c4-9f11-0a78ed90a5d1 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:20.755915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75842759-4008-45c4-9f11-0a78ed90a5d1 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:20.827990 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09710fe-9fb7-44c8-b6ef-04fea2cdff4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:20.829029 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09710fe-9fb7-44c8-b6ef-04fea2cdff4a None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:20.829523 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09710fe-9fb7-44c8-b6ef-04fea2cdff4a None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:20.830008 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09710fe-9fb7-44c8-b6ef-04fea2cdff4a None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:20.831305 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09710fe-9fb7-44c8-b6ef-04fea2cdff4a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:20.831841 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09710fe-9fb7-44c8-b6ef-04fea2cdff4a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:20.834547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09710fe-9fb7-44c8-b6ef-04fea2cdff4a None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:20.857416 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 116/231] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:42:20 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:20.865331 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b2ddef-ae9b-402a-8ae3-fa059ef815f0 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:21.067313 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:21.091152 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b2ddef-ae9b-402a-8ae3-fa059ef815f0 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:21.139516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9197c0c4-f052-43e7-92ac-8cbbdb822420 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:21.140177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9197c0c4-f052-43e7-92ac-8cbbdb822420 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:21.140458 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9197c0c4-f052-43e7-92ac-8cbbdb822420 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:21.140735 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9197c0c4-f052-43e7-92ac-8cbbdb822420 None admin] PATH_INFO: `/v3/users/ba85fdf216fc4cd19bda312f5da53183` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:21.141973 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9197c0c4-f052-43e7-92ac-8cbbdb822420 None admin] RBAC: Authorizing `identity:get_user(user_id=ba85fdf216fc4cd19bda312f5da53183)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:21.143493 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9197c0c4-f052-43e7-92ac-8cbbdb822420 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:21.145380 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 116/232] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:20 2022] GET /identity/v3/users/ba85fdf216fc4cd19bda312f5da53183 => generated 283 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:21.151563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09710fe-9fb7-44c8-b6ef-04fea2cdff4a None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:21.242185 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:21.400239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09710fe-9fb7-44c8-b6ef-04fea2cdff4a None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:21.472008 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab07a09-ff8b-4240-89a3-d2fea65e333e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:21.473071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab07a09-ff8b-4240-89a3-d2fea65e333e None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:21.473560 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab07a09-ff8b-4240-89a3-d2fea65e333e None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:21.474048 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab07a09-ff8b-4240-89a3-d2fea65e333e None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:21.479908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab07a09-ff8b-4240-89a3-d2fea65e333e None admin] RBAC: Authorizing `identity:get_project(project_id=8085b2496c6847aa83e809bde1f3e986)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:21.482750 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab07a09-ff8b-4240-89a3-d2fea65e333e None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:21.485773 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 117/233] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:21 2022] GET /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986 => generated 302 bytes in 335 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:21.491733 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9197c0c4-f052-43e7-92ac-8cbbdb822420 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:21.681002 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9197c0c4-f052-43e7-92ac-8cbbdb822420 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:21.733315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddaff15-bbe7-47f8-a2f5-38573fe5372b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:21.734099 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddaff15-bbe7-47f8-a2f5-38573fe5372b None admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:21.734388 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddaff15-bbe7-47f8-a2f5-38573fe5372b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:21.734695 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddaff15-bbe7-47f8-a2f5-38573fe5372b None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986/users/ba85fdf216fc4cd19bda312f5da53183/roles/cf934cf252ce42d5a0445d3e156348d1` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:21.737171 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddaff15-bbe7-47f8-a2f5-38573fe5372b None admin] RBAC: Authorizing `identity:create_grant(project_id=8085b2496c6847aa83e809bde1f3e986, user_id=ba85fdf216fc4cd19bda312f5da53183, role_id=cf934cf252ce42d5a0445d3e156348d1)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:21.741080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddaff15-bbe7-47f8-a2f5-38573fe5372b None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:21.771666 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 117/234] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:42:21 2022] PUT /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986/users/ba85fdf216fc4cd19bda312f5da53183/roles/cf934cf252ce42d5a0445d3e156348d1 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:42:22.659314 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f39fddf-94e4-4d5c-9574-b59e1b0c02e5 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:22.660134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f39fddf-94e4-4d5c-9574-b59e1b0c02e5 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:22.660134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f39fddf-94e4-4d5c-9574-b59e1b0c02e5 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:22.661239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 118/235] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:22.672437 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21c7ec3-f00c-40ea-a7e6-ffffc401585b None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:22.673195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21c7ec3-f00c-40ea-a7e6-ffffc401585b None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:22.673195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21c7ec3-f00c-40ea-a7e6-ffffc401585b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:22.702725 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-c21c7ec3-f00c-40ea-a7e6-ffffc401585b None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:22.703952 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c21c7ec3-f00c-40ea-a7e6-ffffc401585b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:22.713684 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:22.833096 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 118/236] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Sep 18 00:42:23.016334 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5392716a-a78a-4cf9-bcdf-7ed7abd95785 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:23.016821 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5392716a-a78a-4cf9-bcdf-7ed7abd95785 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:23.017312 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5392716a-a78a-4cf9-bcdf-7ed7abd95785 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:23.020192 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 119/237] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:23 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:23.029491 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ce5b2a-0a22-4f7b-a14e-e94ad00b5b11 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:23.030372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ce5b2a-0a22-4f7b-a14e-e94ad00b5b11 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:23.031131 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ce5b2a-0a22-4f7b-a14e-e94ad00b5b11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:23.062181 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-98ce5b2a-0a22-4f7b-a14e-e94ad00b5b11 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:23.064681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-98ce5b2a-0a22-4f7b-a14e-e94ad00b5b11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:23.171224 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5392716a-a78a-4cf9-bcdf-7ed7abd95785 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:23.212037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 119/238] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:23.404034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5392716a-a78a-4cf9-bcdf-7ed7abd95785 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:23.418683 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:23.472396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd9dfbd-f1c1-40bc-a47e-8a7cad23642c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:23.473304 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd9dfbd-f1c1-40bc-a47e-8a7cad23642c None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:23.473760 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd9dfbd-f1c1-40bc-a47e-8a7cad23642c None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:23.474233 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd9dfbd-f1c1-40bc-a47e-8a7cad23642c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:23.476183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd9dfbd-f1c1-40bc-a47e-8a7cad23642c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:23.479081 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd9dfbd-f1c1-40bc-a47e-8a7cad23642c None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:23.482639 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 120/239] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:42:23 2022] GET /identity/v3/domains/default => generated 199 bytes in 313 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:23.488109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ce5b2a-0a22-4f7b-a14e-e94ad00b5b11 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:23.623633 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:23.625933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3d033d-1f8b-4602-9a18-b1825fc86a85 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:23.626738 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3d033d-1f8b-4602-9a18-b1825fc86a85 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:23.627472 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3d033d-1f8b-4602-9a18-b1825fc86a85 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:23.631520 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 121/240] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:23 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:23.636341 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a268d00c-a578-49d4-acf7-0a4b3f47eb7e None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:23.636686 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a268d00c-a578-49d4-acf7-0a4b3f47eb7e None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:23.636937 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a268d00c-a578-49d4-acf7-0a4b3f47eb7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:23.658710 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-a268d00c-a578-49d4-acf7-0a4b3f47eb7e None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:23.659675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ce5b2a-0a22-4f7b-a14e-e94ad00b5b11 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:23.660329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a268d00c-a578-49d4-acf7-0a4b3f47eb7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:23.702072 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195c1347-18a6-467e-86f8-724eb331c6d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:23.703405 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195c1347-18a6-467e-86f8-724eb331c6d9 None admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:23.703821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195c1347-18a6-467e-86f8-724eb331c6d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:23.704279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195c1347-18a6-467e-86f8-724eb331c6d9 None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:23.705091 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195c1347-18a6-467e-86f8-724eb331c6d9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:23.707060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195c1347-18a6-467e-86f8-724eb331c6d9 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:23.736442 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 120/241] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 18 00:42:23 2022] POST /identity/v3/users => generated 293 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:42:23.737765 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195c1347-18a6-467e-86f8-724eb331c6d9 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:23.748871 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 122/242] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:23.866992 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195c1347-18a6-467e-86f8-724eb331c6d9 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:23.900958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:23.914763 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98b7d34-4f4d-4409-a230-26be49fac23f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:23.915685 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98b7d34-4f4d-4409-a230-26be49fac23f None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:23.916140 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98b7d34-4f4d-4409-a230-26be49fac23f None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:23.916579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98b7d34-4f4d-4409-a230-26be49fac23f None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:23.929950 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98b7d34-4f4d-4409-a230-26be49fac23f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:23.936940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-b98b7d34-4f4d-4409-a230-26be49fac23f None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 18 00:42:23.936940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:23.936940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:23.936940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:23.936940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:23.936940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:23.936940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:23.936940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:23.936940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:23.936940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:23.936940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:23.936940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:23.936940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:23.936940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:23.936940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:23.936940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:23.938135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:23.938135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:23.938135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:23.938135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:23.938135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:23.938135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:23.938135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:23.938135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:23.938135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:23.938135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:23.938135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:23.938135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:23.938135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:23.938135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:23.938135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:23.938135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:23.939242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:23.939242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:23.939242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:23.939242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:23.939242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:23.939242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:23.939242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:23.939242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:23.939242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:23.939242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:23.939242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 18 00:42:23.939242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:42:23.941560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 121/243] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:23 2022] GET /identity/v3/roles/reader => generated 84 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:23.947706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a268d00c-a578-49d4-acf7-0a4b3f47eb7e None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:24.054019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad1307b-ba6d-4641-b489-d31c96f6bfa3 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:24.054651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad1307b-ba6d-4641-b489-d31c96f6bfa3 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:24.055105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad1307b-ba6d-4641-b489-d31c96f6bfa3 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:24.056997 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 122/244] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:24.063207 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708d83d8-de75-4582-ab8a-67fd50bf1a3f None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:24.063740 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708d83d8-de75-4582-ab8a-67fd50bf1a3f None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:24.064189 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708d83d8-de75-4582-ab8a-67fd50bf1a3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:24.092179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-708d83d8-de75-4582-ab8a-67fd50bf1a3f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:24.094852 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-708d83d8-de75-4582-ab8a-67fd50bf1a3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:24.095134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a268d00c-a578-49d4-acf7-0a4b3f47eb7e None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:24.097910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:24.137062 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26234f1-d833-4ce1-9cf6-7f0021f50f34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:24.138225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26234f1-d833-4ce1-9cf6-7f0021f50f34 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:24.138536 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26234f1-d833-4ce1-9cf6-7f0021f50f34 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:24.138813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26234f1-d833-4ce1-9cf6-7f0021f50f34 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:24.139501 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26234f1-d833-4ce1-9cf6-7f0021f50f34 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:24.139804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26234f1-d833-4ce1-9cf6-7f0021f50f34 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:24.141236 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26234f1-d833-4ce1-9cf6-7f0021f50f34 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:24.150950 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 123/245] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:23 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:24.151809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26234f1-d833-4ce1-9cf6-7f0021f50f34 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:24.195218 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 123/246] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:42:24.196539 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708d83d8-de75-4582-ab8a-67fd50bf1a3f None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:24.277254 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26234f1-d833-4ce1-9cf6-7f0021f50f34 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:24.315739 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989a79f4-7185-4c54-af8d-39cf7b5ec22e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:24.316488 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989a79f4-7185-4c54-af8d-39cf7b5ec22e None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:24.316710 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989a79f4-7185-4c54-af8d-39cf7b5ec22e None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:24.316981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989a79f4-7185-4c54-af8d-39cf7b5ec22e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:24.320397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708d83d8-de75-4582-ab8a-67fd50bf1a3f None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:24.329915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-989a79f4-7185-4c54-af8d-39cf7b5ec22e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:24.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-989a79f4-7185-4c54-af8d-39cf7b5ec22e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:24.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:24.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:24.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:24.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:24.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:24.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:24.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:24.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:24.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:24.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:24.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:24.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:24.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:24.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:24.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:24.336259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:24.336259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:24.336259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:24.336259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:24.336259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:24.336259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:24.336259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:24.336259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:24.336259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:24.336259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:24.336259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:24.336259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:24.336259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:24.336259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:24.336259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:24.336259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:24.337053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:24.337053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:24.337053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:24.337053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:24.337053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:24.337053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:24.337053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:24.337053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:24.337053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:24.337053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:24.337053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:24.337053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:42:24.337053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 124/247] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:24 2022] GET /identity/v3/roles/member => generated 84 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:24.342728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989a79f4-7185-4c54-af8d-39cf7b5ec22e None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:24.345377 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:24.368521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501f8a9f-c750-48bf-a7e3-b3514a6805b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:24.369631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501f8a9f-c750-48bf-a7e3-b3514a6805b3 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:24.370201 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501f8a9f-c750-48bf-a7e3-b3514a6805b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:24.370819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501f8a9f-c750-48bf-a7e3-b3514a6805b3 None admin] PATH_INFO: `/v3/users/2ad21cb0f7494b5789c89417df3d8484` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:24.376348 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:24.391346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501f8a9f-c750-48bf-a7e3-b3514a6805b3 None admin] RBAC: Authorizing `identity:get_user(user_id=2ad21cb0f7494b5789c89417df3d8484)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:24.393382 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501f8a9f-c750-48bf-a7e3-b3514a6805b3 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:24.395852 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 124/248] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:24 2022] GET /identity/v3/users/2ad21cb0f7494b5789c89417df3d8484 => generated 297 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:24.401322 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501f8a9f-c750-48bf-a7e3-b3514a6805b3 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:24.460858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989a79f4-7185-4c54-af8d-39cf7b5ec22e None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:24.502480 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ebc7ff-976c-4c14-9866-3b931c643417 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:24.503103 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ebc7ff-976c-4c14-9866-3b931c643417 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:24.503378 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ebc7ff-976c-4c14-9866-3b931c643417 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:24.503648 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ebc7ff-976c-4c14-9866-3b931c643417 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:24.504467 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ebc7ff-976c-4c14-9866-3b931c643417 None admin] RBAC: Adding query filter params (name=member) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:24.504730 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ebc7ff-976c-4c14-9866-3b931c643417 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:24.506396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ebc7ff-976c-4c14-9866-3b931c643417 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:24.516445 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 125/249] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:24 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:24.522344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ebc7ff-976c-4c14-9866-3b931c643417 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:24.523617 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501f8a9f-c750-48bf-a7e3-b3514a6805b3 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:24.571365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8161637e-ffa6-4858-adb0-2d9b9f75c8a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:24.572230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8161637e-ffa6-4858-adb0-2d9b9f75c8a0 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:24.572647 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8161637e-ffa6-4858-adb0-2d9b9f75c8a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:24.573071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8161637e-ffa6-4858-adb0-2d9b9f75c8a0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:24.636843 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:24.649616 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:24.669893 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ebc7ff-976c-4c14-9866-3b931c643417 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:24.677299 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8161637e-ffa6-4858-adb0-2d9b9f75c8a0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:24.679242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8161637e-ffa6-4858-adb0-2d9b9f75c8a0 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:24.719404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a132ec84-a557-4054-bff4-4cc2d9f3820c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:24.720691 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a132ec84-a557-4054-bff4-4cc2d9f3820c None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:24.721227 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a132ec84-a557-4054-bff4-4cc2d9f3820c None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:24.721513 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a132ec84-a557-4054-bff4-4cc2d9f3820c None admin] PATH_INFO: `/v3/users/6e1b221d671c437cb5f87821a5d327ac` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:24.723395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a132ec84-a557-4054-bff4-4cc2d9f3820c None admin] RBAC: Authorizing `identity:get_user(user_id=6e1b221d671c437cb5f87821a5d327ac)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:24.725187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a132ec84-a557-4054-bff4-4cc2d9f3820c None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:24.728603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 126/250] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:24 2022] GET /identity/v3/users/6e1b221d671c437cb5f87821a5d327ac => generated 297 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:24.729531 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a132ec84-a557-4054-bff4-4cc2d9f3820c None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:24.860053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 125/251] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:42:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 459 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:42:24.860670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8161637e-ffa6-4858-adb0-2d9b9f75c8a0 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:24.956701 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a132ec84-a557-4054-bff4-4cc2d9f3820c None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:24.970883 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:25.001672 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8161637e-ffa6-4858-adb0-2d9b9f75c8a0 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:25.043460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683176ba-4dda-4be8-9184-b54ff309522c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:25.045024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683176ba-4dda-4be8-9184-b54ff309522c None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:25.045534 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683176ba-4dda-4be8-9184-b54ff309522c None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:25.046001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683176ba-4dda-4be8-9184-b54ff309522c None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:25.047749 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683176ba-4dda-4be8-9184-b54ff309522c None admin] RBAC: Authorizing `identity:get_project(project_id=8085b2496c6847aa83e809bde1f3e986)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:25.049830 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683176ba-4dda-4be8-9184-b54ff309522c None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:25.051974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 126/252] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:24 2022] GET /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986 => generated 302 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:25.052982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683176ba-4dda-4be8-9184-b54ff309522c None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:25.080485 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e412a1-e26d-447d-981b-5f332eecdbc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:25.081164 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e412a1-e26d-447d-981b-5f332eecdbc2 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:25.081446 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e412a1-e26d-447d-981b-5f332eecdbc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:25.081721 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e412a1-e26d-447d-981b-5f332eecdbc2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:25.131265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e412a1-e26d-447d-981b-5f332eecdbc2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:25.136813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-32e412a1-e26d-447d-981b-5f332eecdbc2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:42:25.136813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:25.136813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:25.136813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:25.136813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:25.136813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:25.136813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:25.136813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:25.136813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:25.136813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:25.136813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:25.136813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:25.136813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:25.136813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:25.136813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:25.136813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:25.137859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:25.137859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:25.137859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:25.137859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:25.137859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:25.137859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:25.137859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:25.137859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:25.137859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:25.137859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:25.137859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:25.137859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:25.137859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:25.137859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:25.137859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:25.137859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:25.138669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:25.138669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:25.138669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:25.138669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:25.138669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:25.138669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:25.138669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:25.138669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:25.138669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:25.138669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:25.138669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:42:25.138669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:42:25.138669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 127/253] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:42:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 409 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:25.139336 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e412a1-e26d-447d-981b-5f332eecdbc2 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:25.197800 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:25.208195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683176ba-4dda-4be8-9184-b54ff309522c None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:25.246435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331cd221-57fa-495c-ab60-3cb47447ba5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:25.248023 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331cd221-57fa-495c-ab60-3cb47447ba5c None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:25.248500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331cd221-57fa-495c-ab60-3cb47447ba5c None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:25.248991 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331cd221-57fa-495c-ab60-3cb47447ba5c None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:25.250544 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331cd221-57fa-495c-ab60-3cb47447ba5c None admin] RBAC: Authorizing `identity:get_project(project_id=8085b2496c6847aa83e809bde1f3e986)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:25.252332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331cd221-57fa-495c-ab60-3cb47447ba5c None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:25.254629 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 127/254] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:25 2022] GET /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986 => generated 302 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:25.255660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331cd221-57fa-495c-ab60-3cb47447ba5c None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:25.271555 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:25.367999 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331cd221-57fa-495c-ab60-3cb47447ba5c None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:25.403672 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e412a1-e26d-447d-981b-5f332eecdbc2 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:25.406316 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011fe8e1-57e1-4eeb-8364-9e8d8d7a1a7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:25.407241 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011fe8e1-57e1-4eeb-8364-9e8d8d7a1a7e None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:25.407740 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011fe8e1-57e1-4eeb-8364-9e8d8d7a1a7e None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:25.408229 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011fe8e1-57e1-4eeb-8364-9e8d8d7a1a7e None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:25.409188 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011fe8e1-57e1-4eeb-8364-9e8d8d7a1a7e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:25.409703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011fe8e1-57e1-4eeb-8364-9e8d8d7a1a7e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:25.411332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011fe8e1-57e1-4eeb-8364-9e8d8d7a1a7e None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:25.420790 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 128/255] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:25 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:25.422007 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011fe8e1-57e1-4eeb-8364-9e8d8d7a1a7e None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:25.472982 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213035ea-0157-4828-9024-b43d5d8e6329 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:25.474019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213035ea-0157-4828-9024-b43d5d8e6329 None admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:25.474539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213035ea-0157-4828-9024-b43d5d8e6329 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:25.475037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213035ea-0157-4828-9024-b43d5d8e6329 None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986/users/6e1b221d671c437cb5f87821a5d327ac/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:25.478857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213035ea-0157-4828-9024-b43d5d8e6329 None admin] RBAC: Authorizing `identity:create_grant(project_id=8085b2496c6847aa83e809bde1f3e986, user_id=6e1b221d671c437cb5f87821a5d327ac, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:25.482447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213035ea-0157-4828-9024-b43d5d8e6329 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:25.505130 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:25.515887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 128/256] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:42:25 2022] PUT /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986/users/6e1b221d671c437cb5f87821a5d327ac/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 377 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:42:25.525277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81311a3d-182f-4b91-9d32-f95a6cbcd203 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:25.525831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81311a3d-182f-4b91-9d32-f95a6cbcd203 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:25.526319 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81311a3d-182f-4b91-9d32-f95a6cbcd203 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:25.533674 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 129/257] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:25 2022] GET /identity => generated 270 bytes in 17 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:25.539668 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81311a3d-182f-4b91-9d32-f95a6cbcd203 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:25.543285 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011fe8e1-57e1-4eeb-8364-9e8d8d7a1a7e None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:25.583650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda69c7a-2d33-41b0-8360-db6621ff8b35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:25.584639 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda69c7a-2d33-41b0-8360-db6621ff8b35 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:25.585137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda69c7a-2d33-41b0-8360-db6621ff8b35 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:25.585643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda69c7a-2d33-41b0-8360-db6621ff8b35 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:25.586637 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda69c7a-2d33-41b0-8360-db6621ff8b35 None admin] RBAC: Adding query filter params (role.id=a928285094b34996a609520c255b4c5d, scope.project.id=8085b2496c6847aa83e809bde1f3e986, user.id=2ad21cb0f7494b5789c89417df3d8484) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:25.587201 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda69c7a-2d33-41b0-8360-db6621ff8b35 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:25.588909 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda69c7a-2d33-41b0-8360-db6621ff8b35 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:25.596189 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 129/258] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:42:25 2022] GET /identity/v3/role_assignments?user.id=2ad21cb0f7494b5789c89417df3d8484&scope.project.id=8085b2496c6847aa83e809bde1f3e986&role.id=a928285094b34996a609520c255b4c5d => generated 259 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:25.597819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2246f5bb-3c39-4722-8508-5542636afc97 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:25.598356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2246f5bb-3c39-4722-8508-5542636afc97 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:25.598872 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2246f5bb-3c39-4722-8508-5542636afc97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:25.620217 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-2246f5bb-3c39-4722-8508-5542636afc97 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:25.621784 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-2246f5bb-3c39-4722-8508-5542636afc97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:25.742296 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 130/259] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:25.769363 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:25.811051 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81311a3d-182f-4b91-9d32-f95a6cbcd203 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:25.864632 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd27ed1b-20c0-45f5-bb34-5c7122a5be8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:25.865615 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd27ed1b-20c0-45f5-bb34-5c7122a5be8a None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:25.865906 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd27ed1b-20c0-45f5-bb34-5c7122a5be8a None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:25.866182 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd27ed1b-20c0-45f5-bb34-5c7122a5be8a None admin] PATH_INFO: `/v3/users/ba85fdf216fc4cd19bda312f5da53183` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:25.867600 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd27ed1b-20c0-45f5-bb34-5c7122a5be8a None admin] RBAC: Authorizing `identity:get_user(user_id=ba85fdf216fc4cd19bda312f5da53183)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:25.869123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd27ed1b-20c0-45f5-bb34-5c7122a5be8a None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:25.870879 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 130/260] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:25 2022] GET /identity/v3/users/ba85fdf216fc4cd19bda312f5da53183 => generated 283 bytes in 337 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:25.877382 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2246f5bb-3c39-4722-8508-5542636afc97 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:25.975915 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:26.012823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2246f5bb-3c39-4722-8508-5542636afc97 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:26.053264 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c83f1ce-9163-4867-ad64-3548febc7af2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:26.053971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c83f1ce-9163-4867-ad64-3548febc7af2 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:26.054254 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c83f1ce-9163-4867-ad64-3548febc7af2 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:26.054554 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c83f1ce-9163-4867-ad64-3548febc7af2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:26.115290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd27ed1b-20c0-45f5-bb34-5c7122a5be8a None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:26.130052 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c83f1ce-9163-4867-ad64-3548febc7af2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:26.131578 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c83f1ce-9163-4867-ad64-3548febc7af2 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:26.203277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:26.244459 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 131/261] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:42:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:42:26.250170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c83f1ce-9163-4867-ad64-3548febc7af2 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:26.359170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c83f1ce-9163-4867-ad64-3548febc7af2 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:26.373477 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd27ed1b-20c0-45f5-bb34-5c7122a5be8a None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:26.411363 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0794989-0b95-4b25-9e86-0baa2498e26d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:26.412057 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0794989-0b95-4b25-9e86-0baa2498e26d None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:26.412337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0794989-0b95-4b25-9e86-0baa2498e26d None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:26.412613 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0794989-0b95-4b25-9e86-0baa2498e26d None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:26.413916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0794989-0b95-4b25-9e86-0baa2498e26d None admin] RBAC: Authorizing `identity:get_project(project_id=8085b2496c6847aa83e809bde1f3e986)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:26.415443 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0794989-0b95-4b25-9e86-0baa2498e26d None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:26.417358 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 132/262] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:26 2022] GET /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986 => generated 302 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:26.423878 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0794989-0b95-4b25-9e86-0baa2498e26d None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:26.471536 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:26.476149 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a296279c-d9d5-413a-897b-bb967953a0b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:26.477244 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a296279c-d9d5-413a-897b-bb967953a0b3 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:26.477757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a296279c-d9d5-413a-897b-bb967953a0b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:26.478588 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a296279c-d9d5-413a-897b-bb967953a0b3 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:26.519670 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a296279c-d9d5-413a-897b-bb967953a0b3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:26.525725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-a296279c-d9d5-413a-897b-bb967953a0b3 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:42:26.525725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:26.525725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:26.525725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:26.525725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:26.525725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:26.525725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:26.525725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:26.525725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:26.525725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:26.525725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:26.525725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:26.525725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:26.525725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:26.525725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:26.525725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:26.526712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:26.526712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:26.526712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:26.526712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:26.526712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:26.526712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:26.526712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:26.526712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:26.526712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:26.526712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:26.526712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:26.526712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:26.526712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:26.526712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:26.526712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:26.526712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:26.527567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:26.527567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:26.527567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:26.527567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:26.527567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:26.527567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:26.527567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:26.527567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:26.527567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:26.527567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:26.527567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:42:26.527567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:42:26.539048 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 131/263] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 18 00:42:26 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 424 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:26.545826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a296279c-d9d5-413a-897b-bb967953a0b3 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:26.553053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:26.568933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0794989-0b95-4b25-9e86-0baa2498e26d None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:26.627491 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb1deab-cdf3-4b04-abdf-53722693a968 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:26.629514 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb1deab-cdf3-4b04-abdf-53722693a968 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:26.629514 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb1deab-cdf3-4b04-abdf-53722693a968 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:26.629514 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb1deab-cdf3-4b04-abdf-53722693a968 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:26.629947 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb1deab-cdf3-4b04-abdf-53722693a968 None admin] RBAC: Adding query filter params (role.id=1e04c9fea65244d28e1e59fc4c64f32f, scope.project.id=8085b2496c6847aa83e809bde1f3e986, user.id=ba85fdf216fc4cd19bda312f5da53183) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:26.630241 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb1deab-cdf3-4b04-abdf-53722693a968 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:26.631785 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb1deab-cdf3-4b04-abdf-53722693a968 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:26.643314 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 133/264] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:42:26 2022] GET /identity/v3/role_assignments?user.id=ba85fdf216fc4cd19bda312f5da53183&scope.project.id=8085b2496c6847aa83e809bde1f3e986&role.id=1e04c9fea65244d28e1e59fc4c64f32f => generated 608 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:26.828896 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a296279c-d9d5-413a-897b-bb967953a0b3 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:26.896944 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a327a67a-58f3-43c3-9c99-39624a7792da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:26.898596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a327a67a-58f3-43c3-9c99-39624a7792da None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:26.899526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a327a67a-58f3-43c3-9c99-39624a7792da None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:26.900239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a327a67a-58f3-43c3-9c99-39624a7792da None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:26.901790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a327a67a-58f3-43c3-9c99-39624a7792da None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:26.902664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a327a67a-58f3-43c3-9c99-39624a7792da None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:26.906273 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a327a67a-58f3-43c3-9c99-39624a7792da None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:26.924859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 132/265] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:42:26 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 379 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:26.931798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb1deab-cdf3-4b04-abdf-53722693a968 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:27.065592 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:27.072212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb1deab-cdf3-4b04-abdf-53722693a968 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:27.123251 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42333b9a-ea2b-49a9-a4b6-1cbcf72d116d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:27.123919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42333b9a-ea2b-49a9-a4b6-1cbcf72d116d None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:27.124205 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42333b9a-ea2b-49a9-a4b6-1cbcf72d116d None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:27.124462 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42333b9a-ea2b-49a9-a4b6-1cbcf72d116d None admin] PATH_INFO: `/v3/users/ba85fdf216fc4cd19bda312f5da53183` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:27.125779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42333b9a-ea2b-49a9-a4b6-1cbcf72d116d None admin] RBAC: Authorizing `identity:get_user(user_id=ba85fdf216fc4cd19bda312f5da53183)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:27.127349 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42333b9a-ea2b-49a9-a4b6-1cbcf72d116d None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:27.129338 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 134/266] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:26 2022] GET /identity/v3/users/ba85fdf216fc4cd19bda312f5da53183 => generated 283 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:27.135540 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a327a67a-58f3-43c3-9c99-39624a7792da None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:27.256064 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:27.311070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a327a67a-58f3-43c3-9c99-39624a7792da None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:27.353829 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2605a9-78e3-437d-8f08-721a816ab2ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:27.354511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2605a9-78e3-437d-8f08-721a816ab2ba None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:27.354785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2605a9-78e3-437d-8f08-721a816ab2ba None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:27.355058 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2605a9-78e3-437d-8f08-721a816ab2ba None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:27.458971 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2605a9-78e3-437d-8f08-721a816ab2ba None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:27.460476 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2605a9-78e3-437d-8f08-721a816ab2ba None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:27.557866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25f8bf4-408a-4f74-8542-415d9d46a521 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:27.558240 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25f8bf4-408a-4f74-8542-415d9d46a521 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:27.558425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25f8bf4-408a-4f74-8542-415d9d46a521 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:27.560676 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 135/267] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:27.564854 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:27.566158 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232f537e-d1ad-40db-a7f6-a7d5d88baa6e None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:27.566439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232f537e-d1ad-40db-a7f6-a7d5d88baa6e None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:27.566744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232f537e-d1ad-40db-a7f6-a7d5d88baa6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:27.599569 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-232f537e-d1ad-40db-a7f6-a7d5d88baa6e None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:27.600789 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-232f537e-d1ad-40db-a7f6-a7d5d88baa6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:27.610342 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 133/268] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:42:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 476 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:42:27.622594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2605a9-78e3-437d-8f08-721a816ab2ba None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:27.753665 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 136/269] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:27.791622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2605a9-78e3-437d-8f08-721a816ab2ba None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:27.844839 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3e4083-cf78-4484-a706-8a781117b5c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:27.845798 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3e4083-cf78-4484-a706-8a781117b5c8 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:27.845798 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3e4083-cf78-4484-a706-8a781117b5c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:27.846060 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3e4083-cf78-4484-a706-8a781117b5c8 None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:27.851334 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3e4083-cf78-4484-a706-8a781117b5c8 None admin] RBAC: Authorizing `identity:get_project(project_id=8085b2496c6847aa83e809bde1f3e986)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:27.852842 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3e4083-cf78-4484-a706-8a781117b5c8 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:27.854613 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 134/270] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:27 2022] GET /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986 => generated 302 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:27.861347 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232f537e-d1ad-40db-a7f6-a7d5d88baa6e None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:27.965766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:27.972058 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232f537e-d1ad-40db-a7f6-a7d5d88baa6e None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:28.010415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a6b037-ae44-4e7f-878d-2a8a8c00fae1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:28.011071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a6b037-ae44-4e7f-878d-2a8a8c00fae1 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:28.011390 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a6b037-ae44-4e7f-878d-2a8a8c00fae1 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:28.011630 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a6b037-ae44-4e7f-878d-2a8a8c00fae1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:28.012356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a6b037-ae44-4e7f-878d-2a8a8c00fae1 None admin] RBAC: Adding query filter params (role.id=cf934cf252ce42d5a0445d3e156348d1, scope.project.id=8085b2496c6847aa83e809bde1f3e986, user.id=ba85fdf216fc4cd19bda312f5da53183) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:28.013217 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a6b037-ae44-4e7f-878d-2a8a8c00fae1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:28.014709 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a6b037-ae44-4e7f-878d-2a8a8c00fae1 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:28.021364 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 137/271] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:42:27 2022] GET /identity/v3/role_assignments?user.id=ba85fdf216fc4cd19bda312f5da53183&scope.project.id=8085b2496c6847aa83e809bde1f3e986&role.id=cf934cf252ce42d5a0445d3e156348d1 => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:28.200697 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3e4083-cf78-4484-a706-8a781117b5c8 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:28.265472 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:28.348625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3e4083-cf78-4484-a706-8a781117b5c8 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:28.393741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f65ddd-1a4a-43b6-ae24-7d60a7553762 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:28.394410 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f65ddd-1a4a-43b6-ae24-7d60a7553762 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:28.394782 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f65ddd-1a4a-43b6-ae24-7d60a7553762 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:28.395064 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f65ddd-1a4a-43b6-ae24-7d60a7553762 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:28.408426 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f65ddd-1a4a-43b6-ae24-7d60a7553762 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:28.413808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-16f65ddd-1a4a-43b6-ae24-7d60a7553762 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:28.413808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:28.413808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:28.413808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:28.413808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:28.413808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:28.413808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:28.413808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:28.413808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:28.413808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:28.413808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:28.413808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:28.413808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:28.413808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:28.413808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:28.413808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:28.415625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:28.415625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:28.415625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:28.415625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:28.415625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:28.415625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:28.415625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:28.415625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:28.415625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:28.415625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:28.415625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:28.415625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:28.415625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:28.415625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:28.415625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:28.415625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:28.417404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:28.417404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:28.417404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:28.417404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:28.417404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:28.417404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:28.417404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:28.417404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:28.417404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:28.417404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:28.417404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:28.417404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:42:28.417404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 135/272] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:28 2022] GET /identity/v3/roles/member => generated 84 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:28.423666 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a6b037-ae44-4e7f-878d-2a8a8c00fae1 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:28.467575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67cc7c2-8803-4870-9a04-ac776376a81f None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:28.467804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67cc7c2-8803-4870-9a04-ac776376a81f None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:28.468107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67cc7c2-8803-4870-9a04-ac776376a81f None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:28.469424 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 136/273] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:28.475324 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f132c3-7443-4119-8866-bc0a75bf46c7 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:28.475719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f132c3-7443-4119-8866-bc0a75bf46c7 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:28.475867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f132c3-7443-4119-8866-bc0a75bf46c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:28.498163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-b8f132c3-7443-4119-8866-bc0a75bf46c7 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:28.499430 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-b8f132c3-7443-4119-8866-bc0a75bf46c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:28.580440 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a6b037-ae44-4e7f-878d-2a8a8c00fae1 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:28.600840 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 137/274] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:42:28.638980 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d16d164-1d1f-4f24-9306-4772d6d7249b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:28.639606 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d16d164-1d1f-4f24-9306-4772d6d7249b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:28.639892 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d16d164-1d1f-4f24-9306-4772d6d7249b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:28.640163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d16d164-1d1f-4f24-9306-4772d6d7249b None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:28.640850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d16d164-1d1f-4f24-9306-4772d6d7249b None admin] RBAC: Adding query filter params (name=member) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:28.641157 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d16d164-1d1f-4f24-9306-4772d6d7249b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:28.642592 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d16d164-1d1f-4f24-9306-4772d6d7249b None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:28.652401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 138/275] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:28 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:28.658217 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f132c3-7443-4119-8866-bc0a75bf46c7 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:28.688527 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:28.800426 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f132c3-7443-4119-8866-bc0a75bf46c7 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:28.870314 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ac7959-8bb6-44d3-9722-ef7b6b1fa8d5 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:28.870644 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ac7959-8bb6-44d3-9722-ef7b6b1fa8d5 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:28.870928 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ac7959-8bb6-44d3-9722-ef7b6b1fa8d5 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:28.873096 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6f247e-d9d9-48ca-9179-18ebbfad3084 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:28.875110 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6f247e-d9d9-48ca-9179-18ebbfad3084 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:28.875880 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6f247e-d9d9-48ca-9179-18ebbfad3084 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:28.876622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6f247e-d9d9-48ca-9179-18ebbfad3084 None admin] PATH_INFO: `/v3/users/9162dc0c569b4340a084ff8306acb893` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:28.879651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 139/276] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:28 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:28.885347 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3dfcbb-d7f7-4e49-8c08-7af0dd493bea None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:28.885693 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3dfcbb-d7f7-4e49-8c08-7af0dd493bea None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:28.885924 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3dfcbb-d7f7-4e49-8c08-7af0dd493bea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:28.926141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6f247e-d9d9-48ca-9179-18ebbfad3084 None admin] RBAC: Authorizing `identity:get_user(user_id=9162dc0c569b4340a084ff8306acb893)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:28.928963 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6f247e-d9d9-48ca-9179-18ebbfad3084 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:28.932103 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 138/277] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:28 2022] GET /identity/v3/users/9162dc0c569b4340a084ff8306acb893 => generated 293 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:28.936401 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6f247e-d9d9-48ca-9179-18ebbfad3084 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:28.937592 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-7a3dfcbb-d7f7-4e49-8c08-7af0dd493bea None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:28.939271 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-7a3dfcbb-d7f7-4e49-8c08-7af0dd493bea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:28.962640 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:29.019430 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 140/278] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:29.021024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3dfcbb-d7f7-4e49-8c08-7af0dd493bea None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:29.103778 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:29.134112 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3dfcbb-d7f7-4e49-8c08-7af0dd493bea None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:29.174933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2684b3f2-f325-475e-a882-851045ff57bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:29.175932 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2684b3f2-f325-475e-a882-851045ff57bd None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:29.176425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2684b3f2-f325-475e-a882-851045ff57bd None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:29.176892 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2684b3f2-f325-475e-a882-851045ff57bd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:29.177849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2684b3f2-f325-475e-a882-851045ff57bd None admin] RBAC: Adding query filter params (role.id=18f6cee478e54567b70d5c7557fe694e, user.id=9162dc0c569b4340a084ff8306acb893, scope.system=all) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:29.178380 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2684b3f2-f325-475e-a882-851045ff57bd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:29.180161 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2684b3f2-f325-475e-a882-851045ff57bd None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:29.187302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 141/279] 10.222.0.42 () {56 vars in 1308 bytes} [Sun Sep 18 00:42:29 2022] GET /identity/v3/role_assignments?user.id=9162dc0c569b4340a084ff8306acb893&scope.system=all&role.id=18f6cee478e54567b70d5c7557fe694e => generated 226 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:29.199178 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6f247e-d9d9-48ca-9179-18ebbfad3084 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:29.247691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6be7f2-0f6d-41f4-b81a-42eb83461da8 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:29.248039 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6be7f2-0f6d-41f4-b81a-42eb83461da8 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:29.248345 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6be7f2-0f6d-41f4-b81a-42eb83461da8 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:29.254427 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 142/280] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:29 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:42:29.255914 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4beb1bec-7d76-4b9b-8e57-ef6cc08936e5 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:29.256195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4beb1bec-7d76-4b9b-8e57-ef6cc08936e5 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:29.256463 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4beb1bec-7d76-4b9b-8e57-ef6cc08936e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:29.275980 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d54e4f-c693-42bb-8731-4cb05273ecaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:29.277068 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d54e4f-c693-42bb-8731-4cb05273ecaf None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:29.277566 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d54e4f-c693-42bb-8731-4cb05273ecaf None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:29.278096 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d54e4f-c693-42bb-8731-4cb05273ecaf None admin] PATH_INFO: `/v3/domains/default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:29.279878 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d54e4f-c693-42bb-8731-4cb05273ecaf None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:29.280630 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-4beb1bec-7d76-4b9b-8e57-ef6cc08936e5 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:29.281921 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-4beb1bec-7d76-4b9b-8e57-ef6cc08936e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:29.282316 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d54e4f-c693-42bb-8731-4cb05273ecaf None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:29.285975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 139/281] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:42:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 353 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:29.292540 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d54e4f-c693-42bb-8731-4cb05273ecaf None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:29.303728 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:29.372363 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 143/282] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:29.432416 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d54e4f-c693-42bb-8731-4cb05273ecaf None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:29.470763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea4a872-183c-4c3e-adc1-e9cd4e87f9f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:29.471665 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea4a872-183c-4c3e-adc1-e9cd4e87f9f2 None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:29.472105 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea4a872-183c-4c3e-adc1-e9cd4e87f9f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:29.472520 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea4a872-183c-4c3e-adc1-e9cd4e87f9f2 None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:29.473335 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea4a872-183c-4c3e-adc1-e9cd4e87f9f2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:29.475034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea4a872-183c-4c3e-adc1-e9cd4e87f9f2 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:29.504567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 140/283] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 18 00:42:29 2022] POST /identity/v3/users => generated 293 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:42:29.558808 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4beb1bec-7d76-4b9b-8e57-ef6cc08936e5 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:29.665418 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea4a872-183c-4c3e-adc1-e9cd4e87f9f2 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:29.678972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:29.683830 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4beb1bec-7d76-4b9b-8e57-ef6cc08936e5 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:29.732519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2987f488-8b85-4fa9-a8c3-0ad46a43300b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:29.733193 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2987f488-8b85-4fa9-a8c3-0ad46a43300b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:29.733471 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2987f488-8b85-4fa9-a8c3-0ad46a43300b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:29.733742 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2987f488-8b85-4fa9-a8c3-0ad46a43300b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:29.749292 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2987f488-8b85-4fa9-a8c3-0ad46a43300b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:29.756384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-2987f488-8b85-4fa9-a8c3-0ad46a43300b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 18 00:42:29.756384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:29.756384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:29.756384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:29.756384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:29.756384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:29.756384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:29.756384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:29.756384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:29.756384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:29.756384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:29.756384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:29.756384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:29.756384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:29.756384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:29.756384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:29.757326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:29.757326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:29.757326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:29.757326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:29.757326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:29.757326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:29.757326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:29.757326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:29.757326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:29.757326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:29.757326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:29.757326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:29.757326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:29.757326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:29.757326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:29.757326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:29.758109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:29.758109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:29.758109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:29.758109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:29.758109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:29.758109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:29.758109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:29.758109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:29.758109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:29.758109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:29.758109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 18 00:42:29.758109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:42:29.758109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 144/284] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:29 2022] GET /identity/v3/roles/reader => generated 84 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:29.769819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2987f488-8b85-4fa9-a8c3-0ad46a43300b None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:29.787082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:29.821366 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea4a872-183c-4c3e-adc1-e9cd4e87f9f2 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:29.881459 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f57daa-d449-41f3-aa4a-24c2272ead4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:29.882108 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f57daa-d449-41f3-aa4a-24c2272ead4a None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:29.882393 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f57daa-d449-41f3-aa4a-24c2272ead4a None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:29.882715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f57daa-d449-41f3-aa4a-24c2272ead4a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:29.902858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f57daa-d449-41f3-aa4a-24c2272ead4a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:29.910402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-c0f57daa-d449-41f3-aa4a-24c2272ead4a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:29.910402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:29.910402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:29.910402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:29.910402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:29.910402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:29.910402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:29.910402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:29.910402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:29.910402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:29.910402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:29.910402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:29.910402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:29.910402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:29.910402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:29.910402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:29.911593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:29.911593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:29.911593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:29.911593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:29.911593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:29.911593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:29.911593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:29.911593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:29.911593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:29.911593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:29.911593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:29.911593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:29.911593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:29.911593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:29.911593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:29.911593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:29.912651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:29.912651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:29.912651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:29.912651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:29.912651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:29.912651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:29.912651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:29.912651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:29.912651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:29.912651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:29.912651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:29.912651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:42:29.914551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 141/285] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:29 2022] GET /identity/v3/roles/member => generated 84 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:29.919060 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f57daa-d449-41f3-aa4a-24c2272ead4a None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:29.975699 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2987f488-8b85-4fa9-a8c3-0ad46a43300b None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:30.020499 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:30.039629 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60a7589-108a-479b-8fa9-8ca5baa0101c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:30.040929 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60a7589-108a-479b-8fa9-8ca5baa0101c None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:30.041441 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60a7589-108a-479b-8fa9-8ca5baa0101c None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:30.041887 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60a7589-108a-479b-8fa9-8ca5baa0101c None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:30.042941 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d60a7589-108a-479b-8fa9-8ca5baa0101c None admin] RBAC: Adding query filter params (name=reader) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:30.043405 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d60a7589-108a-479b-8fa9-8ca5baa0101c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:30.045569 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d60a7589-108a-479b-8fa9-8ca5baa0101c None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:30.056900 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 145/286] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:29 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:30.070105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60a7589-108a-479b-8fa9-8ca5baa0101c None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:30.091208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f57daa-d449-41f3-aa4a-24c2272ead4a None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:30.130530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f58570a-8251-4247-93a1-84ae63258401 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:30.131270 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f58570a-8251-4247-93a1-84ae63258401 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:30.131618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f58570a-8251-4247-93a1-84ae63258401 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:30.131979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f58570a-8251-4247-93a1-84ae63258401 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:30.132771 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f58570a-8251-4247-93a1-84ae63258401 None admin] RBAC: Adding query filter params (name=member) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:30.133145 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f58570a-8251-4247-93a1-84ae63258401 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:30.134720 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f58570a-8251-4247-93a1-84ae63258401 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:30.144922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 142/287] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:29 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:30.150821 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f58570a-8251-4247-93a1-84ae63258401 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:30.165959 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:30.252427 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60a7589-108a-479b-8fa9-8ca5baa0101c None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:30.343022 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010d5586-5b9b-4706-875b-7aa40e6e0304 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:30.343686 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010d5586-5b9b-4706-875b-7aa40e6e0304 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:30.343968 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010d5586-5b9b-4706-875b-7aa40e6e0304 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:30.344245 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010d5586-5b9b-4706-875b-7aa40e6e0304 None admin] PATH_INFO: `/v3/users/2ad21cb0f7494b5789c89417df3d8484` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:30.345685 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010d5586-5b9b-4706-875b-7aa40e6e0304 None admin] RBAC: Authorizing `identity:get_user(user_id=2ad21cb0f7494b5789c89417df3d8484)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:30.350358 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010d5586-5b9b-4706-875b-7aa40e6e0304 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:30.352751 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 146/288] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:30 2022] GET /identity/v3/users/2ad21cb0f7494b5789c89417df3d8484 => generated 297 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:30.361289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010d5586-5b9b-4706-875b-7aa40e6e0304 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:30.374578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f58570a-8251-4247-93a1-84ae63258401 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:30.436413 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:30.459405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218864e4-02f0-464c-9d9e-e636dd7bc499 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:30.460073 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218864e4-02f0-464c-9d9e-e636dd7bc499 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:30.460413 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218864e4-02f0-464c-9d9e-e636dd7bc499 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:30.460682 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218864e4-02f0-464c-9d9e-e636dd7bc499 None admin] PATH_INFO: `/v3/users/6e1b221d671c437cb5f87821a5d327ac` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:30.462028 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-218864e4-02f0-464c-9d9e-e636dd7bc499 None admin] RBAC: Authorizing `identity:get_user(user_id=6e1b221d671c437cb5f87821a5d327ac)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:30.463627 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-218864e4-02f0-464c-9d9e-e636dd7bc499 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:30.465283 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 143/289] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:30 2022] GET /identity/v3/users/6e1b221d671c437cb5f87821a5d327ac => generated 297 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:30.471791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218864e4-02f0-464c-9d9e-e636dd7bc499 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:30.583583 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010d5586-5b9b-4706-875b-7aa40e6e0304 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:30.631213 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b11f52b-1d6f-4517-b431-6d06dde7de7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:30.631965 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b11f52b-1d6f-4517-b431-6d06dde7de7b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:30.632248 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b11f52b-1d6f-4517-b431-6d06dde7de7b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:30.632521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b11f52b-1d6f-4517-b431-6d06dde7de7b None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:30.633747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b11f52b-1d6f-4517-b431-6d06dde7de7b None admin] RBAC: Authorizing `identity:get_project(project_id=8085b2496c6847aa83e809bde1f3e986)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:30.635213 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b11f52b-1d6f-4517-b431-6d06dde7de7b None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:30.637255 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 147/290] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:30 2022] GET /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986 => generated 302 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:30.648543 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b11f52b-1d6f-4517-b431-6d06dde7de7b None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:30.662859 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:30.670750 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218864e4-02f0-464c-9d9e-e636dd7bc499 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:30.738853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9422981-18d6-4b23-9ebe-81acd9e94e2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:30.740098 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9422981-18d6-4b23-9ebe-81acd9e94e2c None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:30.740284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9422981-18d6-4b23-9ebe-81acd9e94e2c None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:30.740689 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9422981-18d6-4b23-9ebe-81acd9e94e2c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:30.861861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b11f52b-1d6f-4517-b431-6d06dde7de7b None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:30.885976 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:30.898040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9422981-18d6-4b23-9ebe-81acd9e94e2c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:30.900538 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9422981-18d6-4b23-9ebe-81acd9e94e2c None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:30.941679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb3ed41-31e0-430f-bb5f-29dafd7f9ca5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:30.942364 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb3ed41-31e0-430f-bb5f-29dafd7f9ca5 None admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:30.942696 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb3ed41-31e0-430f-bb5f-29dafd7f9ca5 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:30.942995 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb3ed41-31e0-430f-bb5f-29dafd7f9ca5 None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986/users/2ad21cb0f7494b5789c89417df3d8484/roles/a928285094b34996a609520c255b4c5d` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:30.946399 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb3ed41-31e0-430f-bb5f-29dafd7f9ca5 None admin] RBAC: Authorizing `identity:create_grant(project_id=8085b2496c6847aa83e809bde1f3e986, user_id=2ad21cb0f7494b5789c89417df3d8484, role_id=a928285094b34996a609520c255b4c5d)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:30.947999 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb3ed41-31e0-430f-bb5f-29dafd7f9ca5 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:30.979286 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 148/291] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:42:30 2022] PUT /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986/users/2ad21cb0f7494b5789c89417df3d8484/roles/a928285094b34996a609520c255b4c5d => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:42:31.122313 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 144/292] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:42:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 650 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:42:31.128382 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb3ed41-31e0-430f-bb5f-29dafd7f9ca5 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:31.228732 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:31.283072 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb3ed41-31e0-430f-bb5f-29dafd7f9ca5 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:31.347935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f7e960-2379-4e62-aa2c-93dfc36a303f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:31.349003 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f7e960-2379-4e62-aa2c-93dfc36a303f None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:31.349487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f7e960-2379-4e62-aa2c-93dfc36a303f None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:31.349968 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f7e960-2379-4e62-aa2c-93dfc36a303f None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:31.351937 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f7e960-2379-4e62-aa2c-93dfc36a303f None admin] RBAC: Authorizing `identity:get_project(project_id=8085b2496c6847aa83e809bde1f3e986)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:31.354723 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f7e960-2379-4e62-aa2c-93dfc36a303f None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:31.358416 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 149/293] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:31 2022] GET /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986 => generated 302 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:31.363961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9422981-18d6-4b23-9ebe-81acd9e94e2c None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:31.503968 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:31.523266 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9422981-18d6-4b23-9ebe-81acd9e94e2c None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:31.590013 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2085d0-8f9a-4548-a893-6eab85b66460 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:31.590747 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2085d0-8f9a-4548-a893-6eab85b66460 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:31.591032 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2085d0-8f9a-4548-a893-6eab85b66460 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:31.591312 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2085d0-8f9a-4548-a893-6eab85b66460 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:31.592082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2085d0-8f9a-4548-a893-6eab85b66460 None admin] RBAC: Adding query filter params (role.id=18f6cee478e54567b70d5c7557fe694e, scope.project.id=8085b2496c6847aa83e809bde1f3e986, user.id=6e1b221d671c437cb5f87821a5d327ac) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:31.592397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2085d0-8f9a-4548-a893-6eab85b66460 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:31.593919 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2085d0-8f9a-4548-a893-6eab85b66460 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:31.604035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 145/294] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:42:31 2022] GET /identity/v3/role_assignments?user.id=6e1b221d671c437cb5f87821a5d327ac&scope.project.id=8085b2496c6847aa83e809bde1f3e986&role.id=18f6cee478e54567b70d5c7557fe694e => generated 608 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:32.315573 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7693e142-a522-4fa7-ac0b-88e3b0d300b9 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:32.316711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7693e142-a522-4fa7-ac0b-88e3b0d300b9 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:32.317224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7693e142-a522-4fa7-ac0b-88e3b0d300b9 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:32.319010 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 150/295] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:32 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:32.325817 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4d41ec-73b4-4978-80c3-dc7e94dca0b4 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:32.326881 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4d41ec-73b4-4978-80c3-dc7e94dca0b4 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:32.327377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4d41ec-73b4-4978-80c3-dc7e94dca0b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:32.363402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-5f4d41ec-73b4-4978-80c3-dc7e94dca0b4 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:32.365977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-5f4d41ec-73b4-4978-80c3-dc7e94dca0b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:32.468860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 146/296] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:32.518970 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4345c76b-3ca2-4887-bb0a-1a9802aff46b None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:32.520089 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4345c76b-3ca2-4887-bb0a-1a9802aff46b None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:32.520588 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4345c76b-3ca2-4887-bb0a-1a9802aff46b None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:32.522168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 151/297] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:32.533401 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6ac582-465d-4053-883a-cbbc3e6b4aa9 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:32.534025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6ac582-465d-4053-883a-cbbc3e6b4aa9 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:32.534560 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6ac582-465d-4053-883a-cbbc3e6b4aa9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:32.555418 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-2b6ac582-465d-4053-883a-cbbc3e6b4aa9 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:32.557007 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-2b6ac582-465d-4053-883a-cbbc3e6b4aa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:32.686723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 147/298] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:32.930282 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4345c76b-3ca2-4887-bb0a-1a9802aff46b None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:33.035740 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4345c76b-3ca2-4887-bb0a-1a9802aff46b None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:33.042186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22c520d-05e9-4f22-a169-ef850b462527 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:33.043141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22c520d-05e9-4f22-a169-ef850b462527 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:33.044397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22c520d-05e9-4f22-a169-ef850b462527 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:33.049157 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 148/299] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:33 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:33.056472 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6865a8db-9123-40cc-a446-508f0718dc96 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:33.057442 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6865a8db-9123-40cc-a446-508f0718dc96 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:33.058105 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6865a8db-9123-40cc-a446-508f0718dc96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:33.070766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:33.077747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aae2751-8721-4fae-b7eb-b724e77d7561 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:33.078392 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae2751-8721-4fae-b7eb-b724e77d7561 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:33.078701 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae2751-8721-4fae-b7eb-b724e77d7561 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:33.078973 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae2751-8721-4fae-b7eb-b724e77d7561 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:33.091186 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aae2751-8721-4fae-b7eb-b724e77d7561 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:33.093625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-6865a8db-9123-40cc-a446-508f0718dc96 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:33.095835 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-6865a8db-9123-40cc-a446-508f0718dc96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:33.096253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-1aae2751-8721-4fae-b7eb-b724e77d7561 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:33.096253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:33.096253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:33.096253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:33.096253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:33.096253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:33.096253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:33.096253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:33.096253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:33.096253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:33.096253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:33.096253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:33.096253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:33.096253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:33.096253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:33.096253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:33.097791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:33.097791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:33.097791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:33.097791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:33.097791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:33.097791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:33.097791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:33.097791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:33.097791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:33.097791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:33.097791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:33.097791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:33.097791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:33.097791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:33.097791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:33.097791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:33.099235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:33.099235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:33.099235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:33.099235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:33.099235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:33.099235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:33.099235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:33.099235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:33.099235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:33.099235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:33.099235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:33.099235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:42:33.099235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 152/300] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:32 2022] GET /identity/v3/roles/member => generated 84 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:42:33.110145 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aae2751-8721-4fae-b7eb-b724e77d7561 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:33.120563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:33.198865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 149/301] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:42:33.201308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6865a8db-9123-40cc-a446-508f0718dc96 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:33.287799 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aae2751-8721-4fae-b7eb-b724e77d7561 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:33.347995 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42982a1b-781e-4262-94db-c5c58d3fb406 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:33.348967 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42982a1b-781e-4262-94db-c5c58d3fb406 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:33.349429 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42982a1b-781e-4262-94db-c5c58d3fb406 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:33.349900 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42982a1b-781e-4262-94db-c5c58d3fb406 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:33.351074 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42982a1b-781e-4262-94db-c5c58d3fb406 None admin] RBAC: Adding query filter params (name=member) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:33.351574 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42982a1b-781e-4262-94db-c5c58d3fb406 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:33.354057 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42982a1b-781e-4262-94db-c5c58d3fb406 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:33.370141 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 153/302] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:33 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:33.376592 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42982a1b-781e-4262-94db-c5c58d3fb406 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:33.411052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6865a8db-9123-40cc-a446-508f0718dc96 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:33.482946 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:33.492335 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ce2f7c-ba09-421c-9c1f-1794cec911dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:33.494087 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ce2f7c-ba09-421c-9c1f-1794cec911dc None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:33.494763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ce2f7c-ba09-421c-9c1f-1794cec911dc None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:33.495187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ce2f7c-ba09-421c-9c1f-1794cec911dc None admin] PATH_INFO: `/v3/domains/default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:33.497277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ce2f7c-ba09-421c-9c1f-1794cec911dc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:33.501211 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ce2f7c-ba09-421c-9c1f-1794cec911dc None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:33.506072 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 150/303] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:42:33 2022] GET /identity/v3/domains/default => generated 199 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:33.518202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ce2f7c-ba09-421c-9c1f-1794cec911dc None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:33.545385 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42982a1b-781e-4262-94db-c5c58d3fb406 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:33.554386 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:33.592628 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bdef5a-3100-443c-848b-040ea668915c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:33.594112 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bdef5a-3100-443c-848b-040ea668915c None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:33.594112 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bdef5a-3100-443c-848b-040ea668915c None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:33.594112 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bdef5a-3100-443c-848b-040ea668915c None admin] PATH_INFO: `/v3/users/9162dc0c569b4340a084ff8306acb893` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:33.595939 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7bdef5a-3100-443c-848b-040ea668915c None admin] RBAC: Authorizing `identity:get_user(user_id=9162dc0c569b4340a084ff8306acb893)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:33.599229 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7bdef5a-3100-443c-848b-040ea668915c None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:33.601175 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 154/304] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:33 2022] GET /identity/v3/users/9162dc0c569b4340a084ff8306acb893 => generated 293 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:33.602490 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bdef5a-3100-443c-848b-040ea668915c None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:33.722497 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ce2f7c-ba09-421c-9c1f-1794cec911dc None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:33.752505 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bdef5a-3100-443c-848b-040ea668915c None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:33.766241 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4200e0a3-2a1a-428d-8b95-2a7960d67f6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:33.766980 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4200e0a3-2a1a-428d-8b95-2a7960d67f6f None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:33.767381 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4200e0a3-2a1a-428d-8b95-2a7960d67f6f None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:33.767738 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4200e0a3-2a1a-428d-8b95-2a7960d67f6f None admin] PATH_INFO: `/v3/groups` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:33.768453 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4200e0a3-2a1a-428d-8b95-2a7960d67f6f None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:33.770191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4200e0a3-2a1a-428d-8b95-2a7960d67f6f None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:33.780660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:33.790358 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 151/305] 10.222.0.42 () {60 vars in 1134 bytes} [Sun Sep 18 00:42:33 2022] POST /identity/v3/groups => generated 229 bytes in 273 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:42:33.791614 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4200e0a3-2a1a-428d-8b95-2a7960d67f6f None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:33.796705 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d089f10-d55c-47b8-841f-4aa5f86d0293 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:33.797366 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d089f10-d55c-47b8-841f-4aa5f86d0293 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:33.797705 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d089f10-d55c-47b8-841f-4aa5f86d0293 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:33.798052 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d089f10-d55c-47b8-841f-4aa5f86d0293 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:33.818421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d089f10-d55c-47b8-841f-4aa5f86d0293 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:33.821895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-5d089f10-d55c-47b8-841f-4aa5f86d0293 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 18 00:42:33.821895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:33.821895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:33.821895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:33.821895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:33.821895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:33.821895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:33.821895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:33.821895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:33.821895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:33.821895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:33.821895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:33.821895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:33.821895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:33.821895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:33.821895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:33.823577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:33.823577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:33.823577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:33.823577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:33.823577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:33.823577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:33.823577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:33.823577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:33.823577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:33.823577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:33.823577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:33.823577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:33.823577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:33.823577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:33.823577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:33.823577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:33.825596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:33.825596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:33.825596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:33.825596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:33.825596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:33.825596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:33.825596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:33.825596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:33.825596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:33.825596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:33.825596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 18 00:42:33.825596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:42:33.832104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 155/306] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:33 2022] GET /identity/v3/roles/reader => generated 84 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:33.833924 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd917e1-06b7-4c36-a012-05aa542ea35c None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:33.834371 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd917e1-06b7-4c36-a012-05aa542ea35c None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:33.834871 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd917e1-06b7-4c36-a012-05aa542ea35c None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:33.837370 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 156/307] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:33 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:33.838311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd917e1-06b7-4c36-a012-05aa542ea35c None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:33.963399 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:34.004076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4200e0a3-2a1a-428d-8b95-2a7960d67f6f None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:34.028334 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd917e1-06b7-4c36-a012-05aa542ea35c None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:34.063088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752ccf5c-fc5d-4162-b8e7-4f8251f15333 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:34.064105 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752ccf5c-fc5d-4162-b8e7-4f8251f15333 None admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:34.064515 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752ccf5c-fc5d-4162-b8e7-4f8251f15333 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:34.064970 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752ccf5c-fc5d-4162-b8e7-4f8251f15333 None admin] PATH_INFO: `/v3/system/users/9162dc0c569b4340a084ff8306acb893/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:34.067437 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752ccf5c-fc5d-4162-b8e7-4f8251f15333 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=9162dc0c569b4340a084ff8306acb893, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:34.069793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752ccf5c-fc5d-4162-b8e7-4f8251f15333 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:34.083037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 152/308] 10.222.0.42 () {58 vars in 1334 bytes} [Sun Sep 18 00:42:33 2022] PUT /identity/v3/system/users/9162dc0c569b4340a084ff8306acb893/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:42:34.084851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f10c664-03e6-4f17-b778-2ae6fb4a6760 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:34.085202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f10c664-03e6-4f17-b778-2ae6fb4a6760 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:34.085677 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f10c664-03e6-4f17-b778-2ae6fb4a6760 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:34.088474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955900ca-a2a5-4dbd-baf5-22ef563bd64b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:34.089226 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955900ca-a2a5-4dbd-baf5-22ef563bd64b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:34.089572 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955900ca-a2a5-4dbd-baf5-22ef563bd64b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:34.089909 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955900ca-a2a5-4dbd-baf5-22ef563bd64b None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:34.090831 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955900ca-a2a5-4dbd-baf5-22ef563bd64b None admin] RBAC: Adding query filter params (name=reader) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:34.091208 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955900ca-a2a5-4dbd-baf5-22ef563bd64b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:34.092871 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955900ca-a2a5-4dbd-baf5-22ef563bd64b None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:34.114509 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 157/309] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:33 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:34.121354 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955900ca-a2a5-4dbd-baf5-22ef563bd64b None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:34.122673 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-9f10c664-03e6-4f17-b778-2ae6fb4a6760 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:34.124246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-9f10c664-03e6-4f17-b778-2ae6fb4a6760 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:34.207302 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 153/310] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:34.223648 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:34.336758 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955900ca-a2a5-4dbd-baf5-22ef563bd64b None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:34.405808 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d19c955-f463-431a-829d-eb74f34b0aac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:34.406902 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d19c955-f463-431a-829d-eb74f34b0aac None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:34.407382 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d19c955-f463-431a-829d-eb74f34b0aac None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:34.407850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d19c955-f463-431a-829d-eb74f34b0aac None admin] PATH_INFO: `/v3/users/33ab6dea02224a0b981481767261f0ca` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:34.435833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d19c955-f463-431a-829d-eb74f34b0aac None admin] RBAC: Authorizing `identity:get_user(user_id=33ab6dea02224a0b981481767261f0ca)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:34.438790 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d19c955-f463-431a-829d-eb74f34b0aac None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:34.442375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 158/311] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:34 2022] GET /identity/v3/users/33ab6dea02224a0b981481767261f0ca => generated 293 bytes in 322 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:34.451942 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f10c664-03e6-4f17-b778-2ae6fb4a6760 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:34.585225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:34.643397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f10c664-03e6-4f17-b778-2ae6fb4a6760 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:34.704493 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d19c955-f463-431a-829d-eb74f34b0aac None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:34.712523 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e63253-438b-4026-9554-4d61b8281bd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:34.713960 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e63253-438b-4026-9554-4d61b8281bd8 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:34.714754 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e63253-438b-4026-9554-4d61b8281bd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:34.715534 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e63253-438b-4026-9554-4d61b8281bd8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:34.717056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e63253-438b-4026-9554-4d61b8281bd8 None admin] RBAC: Adding query filter params (role.id=a928285094b34996a609520c255b4c5d, user.id=33ab6dea02224a0b981481767261f0ca, scope.system=all) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:34.717871 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e63253-438b-4026-9554-4d61b8281bd8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:34.720702 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e63253-438b-4026-9554-4d61b8281bd8 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:34.729790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 154/312] 10.222.0.42 () {56 vars in 1308 bytes} [Sun Sep 18 00:42:34 2022] GET /identity/v3/role_assignments?user.id=33ab6dea02224a0b981481767261f0ca&scope.system=all&role.id=a928285094b34996a609520c255b4c5d => generated 226 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:34.863835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d19c955-f463-431a-829d-eb74f34b0aac None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:34.865727 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:34.911388 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8b6bfe-86b6-4680-b204-6dfbe9b46ce5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:34.911965 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8b6bfe-86b6-4680-b204-6dfbe9b46ce5 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:34.912423 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8b6bfe-86b6-4680-b204-6dfbe9b46ce5 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:34.912706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8b6bfe-86b6-4680-b204-6dfbe9b46ce5 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:34.927276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8b6bfe-86b6-4680-b204-6dfbe9b46ce5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:34.932857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-4a8b6bfe-86b6-4680-b204-6dfbe9b46ce5 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 18 00:42:34.932857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:34.932857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:34.932857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:34.932857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:34.932857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:34.932857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:34.932857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:34.932857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:34.932857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:34.932857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:34.932857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:34.932857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:34.932857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:34.932857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:34.932857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:34.935719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:34.935719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:34.935719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:34.935719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:34.935719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:34.935719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:34.935719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:34.935719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:34.935719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:34.935719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:34.935719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:34.935719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:34.935719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:34.935719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:34.935719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:34.935719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:34.938618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:34.938618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:34.938618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:34.938618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:34.938618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:34.938618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:34.938618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:34.938618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:34.938618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:34.938618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:34.938618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 18 00:42:34.938618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:42:34.938618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 159/313] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:34 2022] GET /identity/v3/roles/reader => generated 84 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:34.950810 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e63253-438b-4026-9554-4d61b8281bd8 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:35.139043 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e63253-438b-4026-9554-4d61b8281bd8 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:35.208239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe73a69-241d-4ea7-8c45-993647405159 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:35.209170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe73a69-241d-4ea7-8c45-993647405159 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:35.209702 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe73a69-241d-4ea7-8c45-993647405159 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:35.210218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe73a69-241d-4ea7-8c45-993647405159 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:35.211407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe73a69-241d-4ea7-8c45-993647405159 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:35.211923 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe73a69-241d-4ea7-8c45-993647405159 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:35.214500 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe73a69-241d-4ea7-8c45-993647405159 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:35.218884 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:35.232602 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 155/314] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:34 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:35.241902 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8b6bfe-86b6-4680-b204-6dfbe9b46ce5 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:35.392349 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8b6bfe-86b6-4680-b204-6dfbe9b46ce5 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:35.430591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb41b4db-e653-4424-b5ba-bd2f5ae4a56b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:35.431593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb41b4db-e653-4424-b5ba-bd2f5ae4a56b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:35.431593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb41b4db-e653-4424-b5ba-bd2f5ae4a56b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:35.431815 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb41b4db-e653-4424-b5ba-bd2f5ae4a56b None admin] PATH_INFO: `/v3/users/2ad21cb0f7494b5789c89417df3d8484` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:35.432919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb41b4db-e653-4424-b5ba-bd2f5ae4a56b None admin] RBAC: Authorizing `identity:get_user(user_id=2ad21cb0f7494b5789c89417df3d8484)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:35.434437 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb41b4db-e653-4424-b5ba-bd2f5ae4a56b None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:35.436403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 160/315] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:35 2022] GET /identity/v3/users/2ad21cb0f7494b5789c89417df3d8484 => generated 297 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:35.445040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe73a69-241d-4ea7-8c45-993647405159 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:35.636896 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe73a69-241d-4ea7-8c45-993647405159 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:35.703577 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b2a899-7959-4b4f-9edd-7f72183829ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:35.704594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b2a899-7959-4b4f-9edd-7f72183829ba None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:35.705079 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b2a899-7959-4b4f-9edd-7f72183829ba None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:35.705557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b2a899-7959-4b4f-9edd-7f72183829ba None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:35.832172 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:35.841023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b2a899-7959-4b4f-9edd-7f72183829ba None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:35.843992 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b2a899-7959-4b4f-9edd-7f72183829ba None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:36.037190 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 156/316] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:42:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 593 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:42:36.045629 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb41b4db-e653-4424-b5ba-bd2f5ae4a56b None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:36.102680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:36.235704 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb41b4db-e653-4424-b5ba-bd2f5ae4a56b None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:36.306394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67551e32-425f-4407-8d42-629239b6b8bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:36.308045 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67551e32-425f-4407-8d42-629239b6b8bb None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:36.308833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67551e32-425f-4407-8d42-629239b6b8bb None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:36.310149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67551e32-425f-4407-8d42-629239b6b8bb None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:36.312577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67551e32-425f-4407-8d42-629239b6b8bb None admin] RBAC: Authorizing `identity:get_project(project_id=8085b2496c6847aa83e809bde1f3e986)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:36.315699 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67551e32-425f-4407-8d42-629239b6b8bb None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:36.319906 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 161/317] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:36 2022] GET /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986 => generated 302 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:36.330693 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b2a899-7959-4b4f-9edd-7f72183829ba None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:36.525113 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:36.531721 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b2a899-7959-4b4f-9edd-7f72183829ba None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:36.602799 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11daa803-5f3c-4f6c-9902-360f55fbdde5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:36.603823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11daa803-5f3c-4f6c-9902-360f55fbdde5 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:36.604310 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11daa803-5f3c-4f6c-9902-360f55fbdde5 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:36.604781 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11daa803-5f3c-4f6c-9902-360f55fbdde5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:36.606195 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11daa803-5f3c-4f6c-9902-360f55fbdde5 None admin] RBAC: Adding query filter params (role.id=a928285094b34996a609520c255b4c5d, scope.project.id=8085b2496c6847aa83e809bde1f3e986, user.id=2ad21cb0f7494b5789c89417df3d8484) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:36.606753 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11daa803-5f3c-4f6c-9902-360f55fbdde5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:36.609980 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11daa803-5f3c-4f6c-9902-360f55fbdde5 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:36.622296 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 157/318] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:42:36 2022] GET /identity/v3/role_assignments?user.id=2ad21cb0f7494b5789c89417df3d8484&scope.project.id=8085b2496c6847aa83e809bde1f3e986&role.id=a928285094b34996a609520c255b4c5d => generated 608 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:36.766781 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ce624f-7f6d-4fcf-ae06-52c2d4ce6d4e None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:36.768400 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ce624f-7f6d-4fcf-ae06-52c2d4ce6d4e None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:36.769210 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ce624f-7f6d-4fcf-ae06-52c2d4ce6d4e None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:36.772603 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 162/319] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:36 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:36.779269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd8a00d-78ca-4541-a240-80dfbc9b526a None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:36.780004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd8a00d-78ca-4541-a240-80dfbc9b526a None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:36.780004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd8a00d-78ca-4541-a240-80dfbc9b526a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:36.810674 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-5bd8a00d-78ca-4541-a240-80dfbc9b526a None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:36.812594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-5bd8a00d-78ca-4541-a240-80dfbc9b526a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:36.934078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 158/320] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:37.102979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e67f6a-47b9-40bb-9dba-6ee4b29ce8fb None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:37.103461 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e67f6a-47b9-40bb-9dba-6ee4b29ce8fb None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:37.103843 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e67f6a-47b9-40bb-9dba-6ee4b29ce8fb None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:37.105386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 163/321] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:37.111335 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d0a609-d0ec-4a99-8586-90dc01d54212 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:37.111769 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d0a609-d0ec-4a99-8586-90dc01d54212 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:37.112163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d0a609-d0ec-4a99-8586-90dc01d54212 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:37.132112 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-89d0a609-d0ec-4a99-8586-90dc01d54212 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:37.133301 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-89d0a609-d0ec-4a99-8586-90dc01d54212 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:37.207271 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e67f6a-47b9-40bb-9dba-6ee4b29ce8fb None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:37.211331 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 159/322] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:42:37.378230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:37.384918 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e67f6a-47b9-40bb-9dba-6ee4b29ce8fb None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:37.426284 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8625de46-0d89-4ad6-a375-17df3b046681 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:37.426957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8625de46-0d89-4ad6-a375-17df3b046681 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:37.427234 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8625de46-0d89-4ad6-a375-17df3b046681 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:37.427501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8625de46-0d89-4ad6-a375-17df3b046681 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:37.440040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8625de46-0d89-4ad6-a375-17df3b046681 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:37.445122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-8625de46-0d89-4ad6-a375-17df3b046681 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:37.445122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:37.445122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:37.445122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:37.445122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:37.445122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:37.445122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:37.445122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:37.445122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:37.445122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:37.445122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:37.445122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:37.445122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:37.445122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:37.445122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:37.445122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:37.447163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:37.447163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:37.447163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:37.447163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:37.447163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:37.447163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:37.447163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:37.447163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:37.447163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:37.447163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:37.447163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:37.447163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:37.447163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:37.447163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:37.447163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:37.447163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:37.449109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:37.449109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:37.449109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:37.449109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:37.449109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:37.449109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:37.449109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:37.449109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:37.449109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:37.449109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:37.449109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:37.449109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:42:37.450820 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 164/323] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:37 2022] GET /identity/v3/roles/member => generated 84 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:37.453501 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d0a609-d0ec-4a99-8586-90dc01d54212 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:37.495308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:37.498153 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8625de46-0d89-4ad6-a375-17df3b046681 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:37.613722 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d0a609-d0ec-4a99-8586-90dc01d54212 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:37.637955 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8625de46-0d89-4ad6-a375-17df3b046681 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:37.664293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e476e720-b4c4-4320-97c4-f336d079a6b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:37.665299 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e476e720-b4c4-4320-97c4-f336d079a6b9 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:37.665744 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e476e720-b4c4-4320-97c4-f336d079a6b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:37.666185 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e476e720-b4c4-4320-97c4-f336d079a6b9 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:37.667338 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e476e720-b4c4-4320-97c4-f336d079a6b9 None admin] RBAC: Adding query filter params (name=member) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:37.667754 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e476e720-b4c4-4320-97c4-f336d079a6b9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:37.670721 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e476e720-b4c4-4320-97c4-f336d079a6b9 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:37.689138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 160/324] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:37 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:37.690694 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e476e720-b4c4-4320-97c4-f336d079a6b9 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:37.697563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bffdb5-7447-446b-9be5-a0e2e6424c00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:37.698588 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bffdb5-7447-446b-9be5-a0e2e6424c00 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:37.699030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bffdb5-7447-446b-9be5-a0e2e6424c00 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:37.699468 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bffdb5-7447-446b-9be5-a0e2e6424c00 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:37.701792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16bffdb5-7447-446b-9be5-a0e2e6424c00 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:37.704301 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16bffdb5-7447-446b-9be5-a0e2e6424c00 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:37.706822 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 165/325] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:42:37 2022] GET /identity/v3/domains/default => generated 199 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:37.712512 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bffdb5-7447-446b-9be5-a0e2e6424c00 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:37.744512 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:37.809345 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:37.832395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e476e720-b4c4-4320-97c4-f336d079a6b9 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:37.838447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bffdb5-7447-446b-9be5-a0e2e6424c00 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:37.888905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01f073a-af78-4f17-a0e8-9befd3b14ded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:37.890034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01f073a-af78-4f17-a0e8-9befd3b14ded None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:37.890501 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01f073a-af78-4f17-a0e8-9befd3b14ded None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:37.890934 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01f073a-af78-4f17-a0e8-9befd3b14ded None admin] PATH_INFO: `/v3/users/9162dc0c569b4340a084ff8306acb893` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:37.892956 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01f073a-af78-4f17-a0e8-9befd3b14ded None admin] RBAC: Authorizing `identity:get_user(user_id=9162dc0c569b4340a084ff8306acb893)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:37.895681 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01f073a-af78-4f17-a0e8-9befd3b14ded None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:37.896928 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a3952e-7e7f-4c50-b82c-38f69fd55815 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:37.897894 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a3952e-7e7f-4c50-b82c-38f69fd55815 None admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:37.898341 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a3952e-7e7f-4c50-b82c-38f69fd55815 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:37.904005 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a3952e-7e7f-4c50-b82c-38f69fd55815 None admin] PATH_INFO: `/v3/groups` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:37.904222 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 161/326] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:37 2022] GET /identity/v3/users/9162dc0c569b4340a084ff8306acb893 => generated 293 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:37.904345 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a3952e-7e7f-4c50-b82c-38f69fd55815 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:37.904345 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a3952e-7e7f-4c50-b82c-38f69fd55815 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:37.906155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01f073a-af78-4f17-a0e8-9befd3b14ded None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:37.938485 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 166/327] 10.222.0.42 () {60 vars in 1134 bytes} [Sun Sep 18 00:42:37 2022] POST /identity/v3/groups => generated 226 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:42:38.000111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305763a9-8e1a-4c64-89d2-ca0344793a40 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:38.000602 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305763a9-8e1a-4c64-89d2-ca0344793a40 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:38.000913 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305763a9-8e1a-4c64-89d2-ca0344793a40 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:38.003875 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 167/328] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:37 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:38.010312 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914ea17d-6f0b-4834-92c0-01397b3c42c4 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:38.010799 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914ea17d-6f0b-4834-92c0-01397b3c42c4 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:38.011223 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914ea17d-6f0b-4834-92c0-01397b3c42c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:38.033423 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-914ea17d-6f0b-4834-92c0-01397b3c42c4 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:38.035187 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-914ea17d-6f0b-4834-92c0-01397b3c42c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:38.056203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01f073a-af78-4f17-a0e8-9befd3b14ded None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:38.103597 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37776a76-8919-4c57-95c1-04eb5e21f4c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:38.104233 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37776a76-8919-4c57-95c1-04eb5e21f4c3 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:38.104511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37776a76-8919-4c57-95c1-04eb5e21f4c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:38.104785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37776a76-8919-4c57-95c1-04eb5e21f4c3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:38.105562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37776a76-8919-4c57-95c1-04eb5e21f4c3 None admin] RBAC: Adding query filter params (role.id=18f6cee478e54567b70d5c7557fe694e, user.id=9162dc0c569b4340a084ff8306acb893, scope.system=all) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:38.105943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37776a76-8919-4c57-95c1-04eb5e21f4c3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:38.108080 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37776a76-8919-4c57-95c1-04eb5e21f4c3 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:38.125550 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 162/329] 10.222.0.42 () {56 vars in 1308 bytes} [Sun Sep 18 00:42:37 2022] GET /identity/v3/role_assignments?user.id=9162dc0c569b4340a084ff8306acb893&scope.system=all&role.id=18f6cee478e54567b70d5c7557fe694e => generated 510 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:38.129954 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 168/330] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:42:38.518197 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37776a76-8919-4c57-95c1-04eb5e21f4c3 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:38.542216 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:38.705841 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37776a76-8919-4c57-95c1-04eb5e21f4c3 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:38.769606 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b37007-108d-45c0-9c2f-a1c381f34a69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:38.771195 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b37007-108d-45c0-9c2f-a1c381f34a69 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:38.771967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b37007-108d-45c0-9c2f-a1c381f34a69 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:38.772717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b37007-108d-45c0-9c2f-a1c381f34a69 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:38.792620 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9b37007-108d-45c0-9c2f-a1c381f34a69 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:38.798275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-f9b37007-108d-45c0-9c2f-a1c381f34a69 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 18 00:42:38.798275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:38.798275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:38.798275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:38.798275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:38.798275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:38.798275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:38.798275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:38.798275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:38.798275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:38.798275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:38.798275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:38.798275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:38.798275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:38.798275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:38.798275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:38.800187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:38.800187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:38.800187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:38.800187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:38.800187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:38.800187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:38.800187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:38.800187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:38.800187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:38.800187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:38.800187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:38.800187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:38.800187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:38.800187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:38.800187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:38.800187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:38.801664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:38.801664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:38.801664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:38.801664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:38.801664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:38.801664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:38.801664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:38.801664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:38.801664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:38.801664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:38.801664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 18 00:42:38.801664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:42:38.801664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 163/331] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:38 2022] GET /identity/v3/roles/reader => generated 84 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:38.813662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914ea17d-6f0b-4834-92c0-01397b3c42c4 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:38.851747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:39.028932 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914ea17d-6f0b-4834-92c0-01397b3c42c4 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:39.098919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae1db11-9ccb-441b-8e98-349b93f8312f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:39.100465 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae1db11-9ccb-441b-8e98-349b93f8312f None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:39.101546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae1db11-9ccb-441b-8e98-349b93f8312f None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:39.102368 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae1db11-9ccb-441b-8e98-349b93f8312f None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:39.103852 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae1db11-9ccb-441b-8e98-349b93f8312f None admin] RBAC: Adding query filter params (name=reader) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:39.104665 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae1db11-9ccb-441b-8e98-349b93f8312f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:39.107906 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae1db11-9ccb-441b-8e98-349b93f8312f None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:39.128716 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 169/332] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:38 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:39.138075 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b37007-108d-45c0-9c2f-a1c381f34a69 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:39.260916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:39.304949 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b37007-108d-45c0-9c2f-a1c381f34a69 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:39.354342 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1212ce-752e-4f33-8e2b-22b4eb030a27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:39.355034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1212ce-752e-4f33-8e2b-22b4eb030a27 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:39.355301 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1212ce-752e-4f33-8e2b-22b4eb030a27 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:39.355569 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1212ce-752e-4f33-8e2b-22b4eb030a27 None admin] PATH_INFO: `/v3/users/33ab6dea02224a0b981481767261f0ca` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:39.358430 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1212ce-752e-4f33-8e2b-22b4eb030a27 None admin] RBAC: Authorizing `identity:get_user(user_id=33ab6dea02224a0b981481767261f0ca)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:39.360022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1212ce-752e-4f33-8e2b-22b4eb030a27 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:39.361685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 164/333] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:39 2022] GET /identity/v3/users/33ab6dea02224a0b981481767261f0ca => generated 293 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:39.371237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae1db11-9ccb-441b-8e98-349b93f8312f None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:39.556067 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:39.606515 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae1db11-9ccb-441b-8e98-349b93f8312f None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:39.681982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8c2ea4-7ebc-4dd5-97ae-275b53ade8ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:39.683155 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8c2ea4-7ebc-4dd5-97ae-275b53ade8ee None admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:39.683641 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8c2ea4-7ebc-4dd5-97ae-275b53ade8ee None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:39.684650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8c2ea4-7ebc-4dd5-97ae-275b53ade8ee None admin] PATH_INFO: `/v3/system/users/33ab6dea02224a0b981481767261f0ca/roles/a928285094b34996a609520c255b4c5d` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:39.691849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8c2ea4-7ebc-4dd5-97ae-275b53ade8ee None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=33ab6dea02224a0b981481767261f0ca, role_id=a928285094b34996a609520c255b4c5d)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:39.694347 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8c2ea4-7ebc-4dd5-97ae-275b53ade8ee None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:39.715050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 170/334] 10.222.0.42 () {58 vars in 1334 bytes} [Sun Sep 18 00:42:39 2022] PUT /identity/v3/system/users/33ab6dea02224a0b981481767261f0ca/roles/a928285094b34996a609520c255b4c5d => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:42:41.408274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c39f63-5d27-4366-975f-5bb1ba254bf4 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:41.409108 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c39f63-5d27-4366-975f-5bb1ba254bf4 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:41.409241 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c39f63-5d27-4366-975f-5bb1ba254bf4 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:41.412282 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 165/335] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:41 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:41.431128 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db3a0fe-eb1e-4820-b647-8eca7753d6ba None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:41.432060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db3a0fe-eb1e-4820-b647-8eca7753d6ba None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:41.432060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db3a0fe-eb1e-4820-b647-8eca7753d6ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:41.459680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-9db3a0fe-eb1e-4820-b647-8eca7753d6ba None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:41.460924 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-9db3a0fe-eb1e-4820-b647-8eca7753d6ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:41.568385 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 171/336] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:42.516439 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c39f63-5d27-4366-975f-5bb1ba254bf4 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:42.710974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c39f63-5d27-4366-975f-5bb1ba254bf4 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:42.712219 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6254e9e4-119f-4995-8f45-4542275f4679 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:42.712855 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6254e9e4-119f-4995-8f45-4542275f4679 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:42.713167 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6254e9e4-119f-4995-8f45-4542275f4679 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:42.719103 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 172/337] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:42 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:42.733731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bc2278-a2c3-43fc-b29c-5bcbca2c933c None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:42.734720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bc2278-a2c3-43fc-b29c-5bcbca2c933c None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:42.735473 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bc2278-a2c3-43fc-b29c-5bcbca2c933c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:42.759225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5a498e-e7f0-4432-a582-eef64e260098 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:42.768251 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5a498e-e7f0-4432-a582-eef64e260098 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:42.769104 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5a498e-e7f0-4432-a582-eef64e260098 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:42.769842 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5a498e-e7f0-4432-a582-eef64e260098 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:42.790146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:42.797603 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-48bc2278-a2c3-43fc-b29c-5bcbca2c933c None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:42.800019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-48bc2278-a2c3-43fc-b29c-5bcbca2c933c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:42.814219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5a498e-e7f0-4432-a582-eef64e260098 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:42.838824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-7d5a498e-e7f0-4432-a582-eef64e260098 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:42.838824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:42.838824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:42.838824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:42.838824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:42.838824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:42.838824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:42.838824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:42.838824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:42.838824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:42.838824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:42.838824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:42.838824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:42.838824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:42.838824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:42.838824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:42.840603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:42.840603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:42.840603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:42.840603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:42.840603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:42.840603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:42.840603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:42.840603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:42.840603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:42.840603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:42.840603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:42.840603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:42.840603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:42.840603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:42.840603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:42.840603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:42.842118 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:42.842118 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:42.842118 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:42.842118 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:42.842118 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:42.842118 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:42.842118 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:42.842118 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:42.842118 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:42.842118 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:42.842118 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:42.842118 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:42:42.842118 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 166/338] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:42 2022] GET /identity/v3/roles/member => generated 84 bytes in 333 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:42.852143 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5a498e-e7f0-4432-a582-eef64e260098 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:43.010445 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 173/339] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 279 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:43.107151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5a498e-e7f0-4432-a582-eef64e260098 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:43.158641 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37745fa0-652a-4471-bb93-76dcc4190354 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:43.159286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37745fa0-652a-4471-bb93-76dcc4190354 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:43.159567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37745fa0-652a-4471-bb93-76dcc4190354 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:43.159850 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37745fa0-652a-4471-bb93-76dcc4190354 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:43.160543 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37745fa0-652a-4471-bb93-76dcc4190354 None admin] RBAC: Adding query filter params (name=member) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:43.160845 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37745fa0-652a-4471-bb93-76dcc4190354 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:43.162279 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37745fa0-652a-4471-bb93-76dcc4190354 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:43.172694 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 167/340] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:42 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:43.179378 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48bc2278-a2c3-43fc-b29c-5bcbca2c933c None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:43.184533 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:43.214314 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba39bb98-4a07-42b4-9e9c-56810eb309fb None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:43.214690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba39bb98-4a07-42b4-9e9c-56810eb309fb None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:43.214892 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba39bb98-4a07-42b4-9e9c-56810eb309fb None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:43.216893 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 168/341] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:42:43.243165 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b543187f-d27c-4de7-af3f-2789c80fffbd None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:43.243510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b543187f-d27c-4de7-af3f-2789c80fffbd None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:43.243718 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b543187f-d27c-4de7-af3f-2789c80fffbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:43.276636 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-b543187f-d27c-4de7-af3f-2789c80fffbd None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:43.278505 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-b543187f-d27c-4de7-af3f-2789c80fffbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:43.409738 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:43.426852 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48bc2278-a2c3-43fc-b29c-5bcbca2c933c None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:43.434549 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 169/342] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:42:43.435939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b7d870-5385-41c1-ab2f-8cd496c33b15 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:43.436278 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b7d870-5385-41c1-ab2f-8cd496c33b15 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:43.436557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b7d870-5385-41c1-ab2f-8cd496c33b15 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:43.439146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 170/343] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:43.443788 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decbdf2f-a657-40bf-8ff2-1ff7bc2c217d None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:43.444075 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decbdf2f-a657-40bf-8ff2-1ff7bc2c217d None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:43.444346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decbdf2f-a657-40bf-8ff2-1ff7bc2c217d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:43.495734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-decbdf2f-a657-40bf-8ff2-1ff7bc2c217d None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:43.497724 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-decbdf2f-a657-40bf-8ff2-1ff7bc2c217d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:43.508467 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655b283a-f497-4335-8110-0a5590e8534c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:43.509471 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655b283a-f497-4335-8110-0a5590e8534c None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:43.509897 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655b283a-f497-4335-8110-0a5590e8534c None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:43.510264 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655b283a-f497-4335-8110-0a5590e8534c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:43.721612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 171/344] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 278 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:42:43.723929 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decbdf2f-a657-40bf-8ff2-1ff7bc2c217d None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:43.725333 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655b283a-f497-4335-8110-0a5590e8534c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:43.727453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655b283a-f497-4335-8110-0a5590e8534c None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:43.734489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:43.945634 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 174/345] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:42:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 767 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:42:43.951581 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655b283a-f497-4335-8110-0a5590e8534c None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:43.989570 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decbdf2f-a657-40bf-8ff2-1ff7bc2c217d None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:44.066950 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:44.106942 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d516552-9876-4ffe-80ce-c0047c4459a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:44.108299 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d516552-9876-4ffe-80ce-c0047c4459a4 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:44.108960 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d516552-9876-4ffe-80ce-c0047c4459a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:44.109627 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d516552-9876-4ffe-80ce-c0047c4459a4 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:44.134301 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d516552-9876-4ffe-80ce-c0047c4459a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:44.147367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-3d516552-9876-4ffe-80ce-c0047c4459a4 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:42:44.147367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:44.147367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:44.147367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:44.147367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:44.147367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:44.147367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:44.147367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:44.147367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:44.147367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:44.147367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:44.147367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:44.147367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:44.147367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:44.147367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:44.147367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:44.149271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:44.149271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:44.149271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:44.149271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:44.149271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:44.149271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:44.149271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:44.149271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:44.149271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:44.149271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:44.149271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:44.149271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:44.149271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:44.149271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:44.149271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:44.149271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:44.151070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:44.151070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:44.151070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:44.151070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:44.151070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:44.151070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:44.151070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:44.151070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:44.151070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:44.151070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:44.151070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:42:44.151070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:42:44.155527 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 172/346] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 18 00:42:43 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 432 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:44.157913 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edf38ce-6c27-4ce6-aca8-8babc04821dc None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:44.158670 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edf38ce-6c27-4ce6-aca8-8babc04821dc None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:44.159502 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edf38ce-6c27-4ce6-aca8-8babc04821dc None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:44.162539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 173/347] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:44 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:44.164100 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edf38ce-6c27-4ce6-aca8-8babc04821dc None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:44.205560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655b283a-f497-4335-8110-0a5590e8534c None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:44.209022 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:44.272060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb91d01-744f-4a92-9a8b-8ca31e71d2c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:44.273032 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb91d01-744f-4a92-9a8b-8ca31e71d2c0 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:44.273450 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb91d01-744f-4a92-9a8b-8ca31e71d2c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:44.273857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb91d01-744f-4a92-9a8b-8ca31e71d2c0 None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:44.275358 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb91d01-744f-4a92-9a8b-8ca31e71d2c0 None admin] RBAC: Authorizing `identity:get_project(project_id=4c46ff2118f948a085f639b37068aa57)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:44.276982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb91d01-744f-4a92-9a8b-8ca31e71d2c0 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:44.279043 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 175/348] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:43 2022] GET /identity/v3/projects/4c46ff2118f948a085f639b37068aa57 => generated 298 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:44.280534 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bff059-5ff2-4475-aaeb-d59b986c27a4 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:44.280991 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bff059-5ff2-4475-aaeb-d59b986c27a4 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:44.281258 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bff059-5ff2-4475-aaeb-d59b986c27a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:44.303393 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-22bff059-5ff2-4475-aaeb-d59b986c27a4 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:44.304976 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-22bff059-5ff2-4475-aaeb-d59b986c27a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:44.394193 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edf38ce-6c27-4ce6-aca8-8babc04821dc None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:44.420867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 176/349] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:44.422084 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bff059-5ff2-4475-aaeb-d59b986c27a4 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:44.476644 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75f7dba-c077-47c5-900d-27ccef1bcfc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:44.478328 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75f7dba-c077-47c5-900d-27ccef1bcfc5 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:44.479148 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75f7dba-c077-47c5-900d-27ccef1bcfc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:44.479881 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75f7dba-c077-47c5-900d-27ccef1bcfc5 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:44.481528 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75f7dba-c077-47c5-900d-27ccef1bcfc5 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:44.482386 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75f7dba-c077-47c5-900d-27ccef1bcfc5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:44.487034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75f7dba-c077-47c5-900d-27ccef1bcfc5 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:44.504466 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 174/350] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:42:44 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 341 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:44.506503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75f7dba-c077-47c5-900d-27ccef1bcfc5 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:44.626376 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:44.645359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:44.652542 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bff059-5ff2-4475-aaeb-d59b986c27a4 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:44.728177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9695b23-922e-47f0-8559-98048211aebd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:44.730130 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9695b23-922e-47f0-8559-98048211aebd None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:44.731004 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9695b23-922e-47f0-8559-98048211aebd None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:44.731716 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9695b23-922e-47f0-8559-98048211aebd None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:44.749713 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75f7dba-c077-47c5-900d-27ccef1bcfc5 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:44.771450 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9695b23-922e-47f0-8559-98048211aebd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:44.781557 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-a9695b23-922e-47f0-8559-98048211aebd None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:42:44.781557 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:44.781557 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:44.781557 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:44.781557 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:44.781557 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:44.781557 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:44.781557 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:44.781557 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:44.781557 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:44.781557 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:44.781557 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:44.781557 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:44.781557 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:44.781557 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:44.781557 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:44.783375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:44.783375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:44.783375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:44.783375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:44.783375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:44.783375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:44.783375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:44.783375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:44.783375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:44.783375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:44.783375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:44.783375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:44.783375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:44.783375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:44.783375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:44.783375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:44.784521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:44.784521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:44.784521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:44.784521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:44.784521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:44.784521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:44.784521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:44.784521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:44.784521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:44.784521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:44.784521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:42:44.784521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:42:44.786013 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 177/351] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 18 00:42:44 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 364 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:44.790499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9695b23-922e-47f0-8559-98048211aebd None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:44.819915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d345ce-b648-479e-a3c4-2f8aff40db1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:44.821012 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d345ce-b648-479e-a3c4-2f8aff40db1c None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:44.821505 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d345ce-b648-479e-a3c4-2f8aff40db1c None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:44.821993 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d345ce-b648-479e-a3c4-2f8aff40db1c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:44.842741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d345ce-b648-479e-a3c4-2f8aff40db1c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:44.848332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-c5d345ce-b648-479e-a3c4-2f8aff40db1c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:44.848332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:44.848332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:44.848332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:44.848332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:44.848332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:44.848332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:44.848332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:44.848332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:44.848332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:44.848332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:44.848332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:44.848332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:44.848332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:44.848332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:44.848332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:44.849719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:44.849719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:44.849719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:44.849719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:44.849719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:44.849719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:44.849719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:44.849719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:44.849719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:44.849719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:44.849719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:44.849719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:44.849719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:44.849719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:44.849719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:44.849719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:44.851056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:44.851056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:44.851056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:44.851056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:44.851056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:44.851056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:44.851056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:44.851056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:44.851056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:44.851056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:44.851056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:44.851056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:42:44.854366 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 175/352] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:44 2022] GET /identity/v3/roles/member => generated 84 bytes in 344 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:44.855446 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d345ce-b648-479e-a3c4-2f8aff40db1c None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:45.014211 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9695b23-922e-47f0-8559-98048211aebd None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:45.045187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d345ce-b648-479e-a3c4-2f8aff40db1c None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:45.048005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:45.076261 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac563963-eb82-4272-aee1-55151e67c107 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:45.077091 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac563963-eb82-4272-aee1-55151e67c107 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:45.077378 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac563963-eb82-4272-aee1-55151e67c107 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:45.077642 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac563963-eb82-4272-aee1-55151e67c107 None admin] PATH_INFO: `/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:45.085442 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac563963-eb82-4272-aee1-55151e67c107 None admin] RBAC: Authorizing `identity:get_group(group_id=cde72a6c511d455a8d0c64ae4d97ef26)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:45.087290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac563963-eb82-4272-aee1-55151e67c107 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:45.089682 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 178/353] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 18 00:42:44 2022] GET /identity/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26 => generated 226 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:45.091148 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac563963-eb82-4272-aee1-55151e67c107 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:45.127944 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78528dad-cfb4-4d56-8a78-aff2ffef6f9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:45.128919 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78528dad-cfb4-4d56-8a78-aff2ffef6f9e None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:45.129795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78528dad-cfb4-4d56-8a78-aff2ffef6f9e None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:45.129952 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78528dad-cfb4-4d56-8a78-aff2ffef6f9e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:45.151058 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:45.270764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78528dad-cfb4-4d56-8a78-aff2ffef6f9e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:45.271393 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78528dad-cfb4-4d56-8a78-aff2ffef6f9e None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:45.317274 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac563963-eb82-4272-aee1-55151e67c107 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:45.409188 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7196c9d7-f02d-4b94-b756-64fcb9f9dcc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:45.410014 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7196c9d7-f02d-4b94-b756-64fcb9f9dcc1 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:45.410631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7196c9d7-f02d-4b94-b756-64fcb9f9dcc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:45.411914 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7196c9d7-f02d-4b94-b756-64fcb9f9dcc1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:45.438730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7196c9d7-f02d-4b94-b756-64fcb9f9dcc1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:45.450136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-7196c9d7-f02d-4b94-b756-64fcb9f9dcc1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:42:45.450136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:45.450136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:45.450136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:45.450136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:45.450136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:45.450136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:45.450136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:45.450136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:45.450136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:45.450136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:45.450136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:45.450136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:45.450136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:45.450136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:45.450136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:45.451136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:45.451136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:45.451136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:45.451136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:45.451136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:45.451136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:45.451136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:45.451136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:45.451136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:45.451136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:45.451136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:45.451136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:45.451136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:45.451136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:45.451136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:45.451136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:45.451944 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:45.451944 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:45.451944 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:45.451944 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:45.451944 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:45.451944 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:45.451944 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:45.451944 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:45.451944 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:45.451944 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:45.451944 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:42:45.451944 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:42:45.452550 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 179/354] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:42:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 362 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:45.455525 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7196c9d7-f02d-4b94-b756-64fcb9f9dcc1 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:45.481988 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 176/355] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:42:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 627 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:42:45.483233 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78528dad-cfb4-4d56-8a78-aff2ffef6f9e None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:45.486006 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:45.604566 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:45.637311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7196c9d7-f02d-4b94-b756-64fcb9f9dcc1 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:45.651828 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78528dad-cfb4-4d56-8a78-aff2ffef6f9e None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:45.705594 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132501e6-6e9d-4e80-bd53-33ef141daf80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:45.706954 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132501e6-6e9d-4e80-bd53-33ef141daf80 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:45.707973 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132501e6-6e9d-4e80-bd53-33ef141daf80 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:45.708488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132501e6-6e9d-4e80-bd53-33ef141daf80 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:45.710424 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-132501e6-6e9d-4e80-bd53-33ef141daf80 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:45.710602 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-132501e6-6e9d-4e80-bd53-33ef141daf80 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:45.713029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-132501e6-6e9d-4e80-bd53-33ef141daf80 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:45.718873 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d30db19-27c6-442b-bc03-f3f35409aa07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:45.719969 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d30db19-27c6-442b-bc03-f3f35409aa07 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:45.720476 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d30db19-27c6-442b-bc03-f3f35409aa07 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:45.720973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d30db19-27c6-442b-bc03-f3f35409aa07 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:45.722241 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d30db19-27c6-442b-bc03-f3f35409aa07 None admin] RBAC: Adding query filter params (name=member) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:45.724318 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d30db19-27c6-442b-bc03-f3f35409aa07 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:45.726994 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d30db19-27c6-442b-bc03-f3f35409aa07 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:45.729781 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 180/356] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:42:45 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:45.732534 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132501e6-6e9d-4e80-bd53-33ef141daf80 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:45.742808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 177/357] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:45 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:45.745179 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e476d9-462b-47aa-b453-6668c0cbb19c None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:45.745690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e476d9-462b-47aa-b453-6668c0cbb19c None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:45.746191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e476d9-462b-47aa-b453-6668c0cbb19c None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:45.748951 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 178/358] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:45 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:45.750407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e476d9-462b-47aa-b453-6668c0cbb19c None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:45.884445 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:45.946647 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132501e6-6e9d-4e80-bd53-33ef141daf80 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:45.956353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e476d9-462b-47aa-b453-6668c0cbb19c None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:46.023216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9364f225-ff5f-44f5-9ea4-a5cf0895fe3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:46.024182 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9364f225-ff5f-44f5-9ea4-a5cf0895fe3d None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:46.024609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9364f225-ff5f-44f5-9ea4-a5cf0895fe3d None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:46.025025 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9364f225-ff5f-44f5-9ea4-a5cf0895fe3d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:46.026288 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9364f225-ff5f-44f5-9ea4-a5cf0895fe3d None admin] RBAC: Adding query filter params (group.id=cde72a6c511d455a8d0c64ae4d97ef26, role.id=18f6cee478e54567b70d5c7557fe694e, scope.project.id=4c46ff2118f948a085f639b37068aa57) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:46.030508 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9364f225-ff5f-44f5-9ea4-a5cf0895fe3d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:46.031113 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9364f225-ff5f-44f5-9ea4-a5cf0895fe3d None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:46.032461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d091da18-5339-4740-b104-c402ef38fee4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:46.033428 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d091da18-5339-4740-b104-c402ef38fee4 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:46.033897 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d091da18-5339-4740-b104-c402ef38fee4 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:46.034315 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d091da18-5339-4740-b104-c402ef38fee4 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:46.037131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d091da18-5339-4740-b104-c402ef38fee4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:46.037601 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d091da18-5339-4740-b104-c402ef38fee4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:46.040093 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d091da18-5339-4740-b104-c402ef38fee4 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:46.046048 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 181/359] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Sep 18 00:42:45 2022] GET /identity/v3/role_assignments?group.id=cde72a6c511d455a8d0c64ae4d97ef26&scope.project.id=4c46ff2118f948a085f639b37068aa57&role.id=18f6cee478e54567b70d5c7557fe694e => generated 260 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:46.048065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9364f225-ff5f-44f5-9ea4-a5cf0895fe3d None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:46.061393 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 179/360] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:46.063017 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d091da18-5339-4740-b104-c402ef38fee4 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:46.071096 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:46.241385 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9364f225-ff5f-44f5-9ea4-a5cf0895fe3d None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:46.279168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d091da18-5339-4740-b104-c402ef38fee4 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:46.286174 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87947dd-5222-4b40-adb6-5fd5860e4c80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:46.286878 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87947dd-5222-4b40-adb6-5fd5860e4c80 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:46.287154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87947dd-5222-4b40-adb6-5fd5860e4c80 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:46.287421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87947dd-5222-4b40-adb6-5fd5860e4c80 None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:46.288721 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87947dd-5222-4b40-adb6-5fd5860e4c80 None admin] RBAC: Authorizing `identity:get_project(project_id=4c46ff2118f948a085f639b37068aa57)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:46.290225 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87947dd-5222-4b40-adb6-5fd5860e4c80 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:46.300101 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 182/361] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:46 2022] GET /identity/v3/projects/4c46ff2118f948a085f639b37068aa57 => generated 298 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:46.300101 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87947dd-5222-4b40-adb6-5fd5860e4c80 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:46.335675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:46.352223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32276f2-d802-49b0-9347-f22d0d42bdb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:46.353916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32276f2-d802-49b0-9347-f22d0d42bdb7 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:46.353916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32276f2-d802-49b0-9347-f22d0d42bdb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:46.354176 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32276f2-d802-49b0-9347-f22d0d42bdb7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:46.440781 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:46.482087 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87947dd-5222-4b40-adb6-5fd5860e4c80 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:46.503531 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32276f2-d802-49b0-9347-f22d0d42bdb7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:46.506258 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32276f2-d802-49b0-9347-f22d0d42bdb7 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:46.554383 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345b059d-edec-48f6-9005-793fb967f179 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:46.555471 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345b059d-edec-48f6-9005-793fb967f179 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:46.555967 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345b059d-edec-48f6-9005-793fb967f179 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:46.556452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345b059d-edec-48f6-9005-793fb967f179 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:46.695484 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 180/362] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:42:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 633 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:42:46.697795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8d19a5-b321-46b8-a623-eb263be23b85 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:46.698431 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8d19a5-b321-46b8-a623-eb263be23b85 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:46.699085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8d19a5-b321-46b8-a623-eb263be23b85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:46.703250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-345b059d-edec-48f6-9005-793fb967f179 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:46.705738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-345b059d-edec-48f6-9005-793fb967f179 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:46.730041 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-df8d19a5-b321-46b8-a623-eb263be23b85 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:46.732533 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-df8d19a5-b321-46b8-a623-eb263be23b85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:46.768519 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:46.846807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:46.847848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 181/363] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:46.849338 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8d19a5-b321-46b8-a623-eb263be23b85 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:46.915339 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 183/364] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:42:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 622 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:42:46.917349 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345b059d-edec-48f6-9005-793fb967f179 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:47.062056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8d19a5-b321-46b8-a623-eb263be23b85 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:47.132008 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140af30c-8e73-4ecd-ad78-12382d826bad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:47.132866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140af30c-8e73-4ecd-ad78-12382d826bad None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:47.133431 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140af30c-8e73-4ecd-ad78-12382d826bad None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:47.133996 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140af30c-8e73-4ecd-ad78-12382d826bad None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:47.156605 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345b059d-edec-48f6-9005-793fb967f179 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:47.183307 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:47.200141 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc583df-6f32-4e3e-aeb3-0141e83e0f37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:47.200784 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc583df-6f32-4e3e-aeb3-0141e83e0f37 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:47.201051 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc583df-6f32-4e3e-aeb3-0141e83e0f37 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:47.201321 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc583df-6f32-4e3e-aeb3-0141e83e0f37 None admin] PATH_INFO: `/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:47.202494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc583df-6f32-4e3e-aeb3-0141e83e0f37 None admin] RBAC: Authorizing `identity:get_group(group_id=cde72a6c511d455a8d0c64ae4d97ef26)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:47.204151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc583df-6f32-4e3e-aeb3-0141e83e0f37 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:47.206098 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 184/365] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 18 00:42:46 2022] GET /identity/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26 => generated 226 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:47.207967 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc583df-6f32-4e3e-aeb3-0141e83e0f37 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:47.259339 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:47.273287 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-140af30c-8e73-4ecd-ad78-12382d826bad None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:47.276488 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-140af30c-8e73-4ecd-ad78-12382d826bad None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:47.322575 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc583df-6f32-4e3e-aeb3-0141e83e0f37 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:47.363389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80924953-dec4-4e2a-b512-fd307f76dc7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:47.364357 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80924953-dec4-4e2a-b512-fd307f76dc7e None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:47.364805 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80924953-dec4-4e2a-b512-fd307f76dc7e None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:47.365229 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80924953-dec4-4e2a-b512-fd307f76dc7e None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:47.366846 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80924953-dec4-4e2a-b512-fd307f76dc7e None admin] RBAC: Authorizing `identity:get_project(project_id=8085b2496c6847aa83e809bde1f3e986)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:47.368993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80924953-dec4-4e2a-b512-fd307f76dc7e None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:47.371533 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 185/366] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:47 2022] GET /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986 => generated 302 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:47.372879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80924953-dec4-4e2a-b512-fd307f76dc7e None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:47.429271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 182/367] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:42:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 581 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:42:47.430523 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140af30c-8e73-4ecd-ad78-12382d826bad None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:47.504880 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:47.511461 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80924953-dec4-4e2a-b512-fd307f76dc7e None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:47.533562 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:47.562905 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e4286b-339d-4b8e-b4b1-cd3367ae3362 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:47.563996 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e4286b-339d-4b8e-b4b1-cd3367ae3362 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:47.564455 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e4286b-339d-4b8e-b4b1-cd3367ae3362 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:47.564916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e4286b-339d-4b8e-b4b1-cd3367ae3362 None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:47.566817 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e4286b-339d-4b8e-b4b1-cd3367ae3362 None admin] RBAC: Authorizing `identity:get_project(project_id=8085b2496c6847aa83e809bde1f3e986)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:47.569189 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e4286b-339d-4b8e-b4b1-cd3367ae3362 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:47.572437 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 186/368] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:47 2022] GET /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986 => generated 302 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:47.573928 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e4286b-339d-4b8e-b4b1-cd3367ae3362 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:47.584504 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140af30c-8e73-4ecd-ad78-12382d826bad None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:47.630418 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66500f0-c05f-4111-aef2-a3d369b8830c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:47.631418 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66500f0-c05f-4111-aef2-a3d369b8830c None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:47.631894 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66500f0-c05f-4111-aef2-a3d369b8830c None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:47.632365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66500f0-c05f-4111-aef2-a3d369b8830c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:47.633661 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66500f0-c05f-4111-aef2-a3d369b8830c None admin] RBAC: Adding query filter params (group.id=cde72a6c511d455a8d0c64ae4d97ef26, role.id=cf934cf252ce42d5a0445d3e156348d1, scope.project.id=4c46ff2118f948a085f639b37068aa57) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:47.634511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66500f0-c05f-4111-aef2-a3d369b8830c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:47.636342 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66500f0-c05f-4111-aef2-a3d369b8830c None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:47.645182 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 183/369] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Sep 18 00:42:47 2022] GET /identity/v3/role_assignments?group.id=cde72a6c511d455a8d0c64ae4d97ef26&scope.project.id=4c46ff2118f948a085f639b37068aa57&role.id=cf934cf252ce42d5a0445d3e156348d1 => generated 260 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:47.645182 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66500f0-c05f-4111-aef2-a3d369b8830c None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:47.774021 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e4286b-339d-4b8e-b4b1-cd3367ae3362 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:47.842237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6d64d2-c3b1-4ba0-9225-27f9f9ff9da8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:47.842997 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6d64d2-c3b1-4ba0-9225-27f9f9ff9da8 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:47.844117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6d64d2-c3b1-4ba0-9225-27f9f9ff9da8 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:47.844567 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6d64d2-c3b1-4ba0-9225-27f9f9ff9da8 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:47.869091 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6d64d2-c3b1-4ba0-9225-27f9f9ff9da8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:47.871805 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66500f0-c05f-4111-aef2-a3d369b8830c None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:47.880622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-8e6d64d2-c3b1-4ba0-9225-27f9f9ff9da8 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 18 00:42:47.880622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:47.880622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:47.880622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:47.880622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:47.880622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:47.880622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:47.880622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:47.880622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:47.880622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:47.880622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:47.880622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:47.880622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:47.880622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:47.880622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:47.880622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:47.881600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:47.881600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:47.881600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:47.881600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:47.881600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:47.881600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:47.881600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:47.881600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:47.881600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:47.881600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:47.881600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:47.881600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:47.881600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:47.881600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:47.881600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:47.881600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:47.882384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:47.882384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:47.882384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:47.882384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:47.882384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:47.882384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:47.882384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:47.882384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:47.882384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:47.882384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:47.882384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 18 00:42:47.882384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:42:47.884621 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 187/370] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:47 2022] GET /identity/v3/roles/reader => generated 84 bytes in 310 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:47.884621 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6d64d2-c3b1-4ba0-9225-27f9f9ff9da8 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:47.915314 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:47.946326 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeea93e3-20ab-4857-98be-2818ce61f548 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:47.948038 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeea93e3-20ab-4857-98be-2818ce61f548 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:47.948374 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeea93e3-20ab-4857-98be-2818ce61f548 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:47.948873 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeea93e3-20ab-4857-98be-2818ce61f548 None admin] PATH_INFO: `/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:47.954137 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeea93e3-20ab-4857-98be-2818ce61f548 None admin] RBAC: Authorizing `identity:get_group(group_id=cde72a6c511d455a8d0c64ae4d97ef26)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:47.955039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeea93e3-20ab-4857-98be-2818ce61f548 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:47.959509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 184/371] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 18 00:42:47 2022] GET /identity/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26 => generated 226 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:47.961053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeea93e3-20ab-4857-98be-2818ce61f548 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:47.978792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:48.071400 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6d64d2-c3b1-4ba0-9225-27f9f9ff9da8 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:48.137761 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046b47d7-861b-43f7-b2ce-17ff81cd14d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:48.138811 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046b47d7-861b-43f7-b2ce-17ff81cd14d6 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:48.139351 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046b47d7-861b-43f7-b2ce-17ff81cd14d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:48.139903 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046b47d7-861b-43f7-b2ce-17ff81cd14d6 None admin] PATH_INFO: `/v3/projects/2e41e0644d514666af8ca7ed05ebaec8` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:48.143570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046b47d7-861b-43f7-b2ce-17ff81cd14d6 None admin] RBAC: Authorizing `identity:get_project(project_id=2e41e0644d514666af8ca7ed05ebaec8)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:48.147964 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046b47d7-861b-43f7-b2ce-17ff81cd14d6 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:48.152779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 188/372] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:47 2022] GET /identity/v3/projects/2e41e0644d514666af8ca7ed05ebaec8 => generated 344 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:48.155282 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046b47d7-861b-43f7-b2ce-17ff81cd14d6 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:48.157715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeea93e3-20ab-4857-98be-2818ce61f548 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:48.202136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6956dff3-b172-469d-80dd-19918955df42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:48.202890 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6956dff3-b172-469d-80dd-19918955df42 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:48.203169 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6956dff3-b172-469d-80dd-19918955df42 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:48.203440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6956dff3-b172-469d-80dd-19918955df42 None admin] PATH_INFO: `/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:48.204698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6956dff3-b172-469d-80dd-19918955df42 None admin] RBAC: Authorizing `identity:get_group(group_id=cde72a6c511d455a8d0c64ae4d97ef26)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:48.206324 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6956dff3-b172-469d-80dd-19918955df42 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:48.208311 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 185/373] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 18 00:42:47 2022] GET /identity/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26 => generated 226 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:48.208946 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6956dff3-b172-469d-80dd-19918955df42 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:48.320078 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046b47d7-861b-43f7-b2ce-17ff81cd14d6 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:48.338222 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:48.345559 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:48.375115 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a806f8fa-fc47-446c-9ce5-62f88e6b1a28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:48.375783 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a806f8fa-fc47-446c-9ce5-62f88e6b1a28 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:48.376145 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a806f8fa-fc47-446c-9ce5-62f88e6b1a28 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:48.376422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a806f8fa-fc47-446c-9ce5-62f88e6b1a28 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:48.377126 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a806f8fa-fc47-446c-9ce5-62f88e6b1a28 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:48.377410 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a806f8fa-fc47-446c-9ce5-62f88e6b1a28 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:48.378838 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a806f8fa-fc47-446c-9ce5-62f88e6b1a28 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:48.388589 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 189/374] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:48 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:48.389354 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a806f8fa-fc47-446c-9ce5-62f88e6b1a28 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:48.439668 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6956dff3-b172-469d-80dd-19918955df42 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:48.501114 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a806f8fa-fc47-446c-9ce5-62f88e6b1a28 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:48.507562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6fc744-22c6-4c7e-b0be-af56e4b44f83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:48.513933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6fc744-22c6-4c7e-b0be-af56e4b44f83 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:48.514371 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6fc744-22c6-4c7e-b0be-af56e4b44f83 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:48.514832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6fc744-22c6-4c7e-b0be-af56e4b44f83 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:48.515947 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6fc744-22c6-4c7e-b0be-af56e4b44f83 None admin] RBAC: Adding query filter params (group.id=cde72a6c511d455a8d0c64ae4d97ef26, role.id=cf934cf252ce42d5a0445d3e156348d1, scope.project.id=8085b2496c6847aa83e809bde1f3e986) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:48.516385 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6fc744-22c6-4c7e-b0be-af56e4b44f83 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:48.521715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6fc744-22c6-4c7e-b0be-af56e4b44f83 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:48.532891 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 186/375] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Sep 18 00:42:48 2022] GET /identity/v3/role_assignments?group.id=cde72a6c511d455a8d0c64ae4d97ef26&scope.project.id=8085b2496c6847aa83e809bde1f3e986&role.id=cf934cf252ce42d5a0445d3e156348d1 => generated 260 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:48.534371 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6fc744-22c6-4c7e-b0be-af56e4b44f83 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:48.562501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230ac140-01fb-4587-8226-bae1e193335c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:48.563570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230ac140-01fb-4587-8226-bae1e193335c None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:48.564068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230ac140-01fb-4587-8226-bae1e193335c None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:48.564582 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230ac140-01fb-4587-8226-bae1e193335c None admin] PATH_INFO: `/v3/groups/6af07276afdc4cebab987faea681089d` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:48.575185 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230ac140-01fb-4587-8226-bae1e193335c None admin] RBAC: Authorizing `identity:get_group(group_id=6af07276afdc4cebab987faea681089d)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:48.577143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230ac140-01fb-4587-8226-bae1e193335c None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:48.579168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 190/376] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 18 00:42:48 2022] GET /identity/v3/groups/6af07276afdc4cebab987faea681089d => generated 229 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:48.580212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230ac140-01fb-4587-8226-bae1e193335c None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:48.662923 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:48.757045 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6fc744-22c6-4c7e-b0be-af56e4b44f83 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:48.813511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:48.815822 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230ac140-01fb-4587-8226-bae1e193335c None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:48.851181 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015b846b-d459-42b4-b768-d638963f1753 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:48.852158 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015b846b-d459-42b4-b768-d638963f1753 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:48.852608 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015b846b-d459-42b4-b768-d638963f1753 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:48.853048 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015b846b-d459-42b4-b768-d638963f1753 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:48.854673 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-015b846b-d459-42b4-b768-d638963f1753 None admin] RBAC: Adding query filter params (group.id=cde72a6c511d455a8d0c64ae4d97ef26, role.id=18f6cee478e54567b70d5c7557fe694e, scope.project.id=8085b2496c6847aa83e809bde1f3e986) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:48.855173 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-015b846b-d459-42b4-b768-d638963f1753 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:48.858298 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-015b846b-d459-42b4-b768-d638963f1753 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:48.870792 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 187/377] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Sep 18 00:42:48 2022] GET /identity/v3/role_assignments?group.id=cde72a6c511d455a8d0c64ae4d97ef26&scope.project.id=8085b2496c6847aa83e809bde1f3e986&role.id=18f6cee478e54567b70d5c7557fe694e => generated 260 bytes in 337 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:48.872766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015b846b-d459-42b4-b768-d638963f1753 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:48.883990 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85390e0-7451-4cc3-9a77-45fb5d2771df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:48.885013 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85390e0-7451-4cc3-9a77-45fb5d2771df None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:48.885461 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85390e0-7451-4cc3-9a77-45fb5d2771df None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:48.885930 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85390e0-7451-4cc3-9a77-45fb5d2771df None admin] PATH_INFO: `/v3/users/33ab6dea02224a0b981481767261f0ca` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:48.888189 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85390e0-7451-4cc3-9a77-45fb5d2771df None admin] RBAC: Authorizing `identity:get_user(user_id=33ab6dea02224a0b981481767261f0ca)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:48.890839 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85390e0-7451-4cc3-9a77-45fb5d2771df None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:48.894376 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 191/378] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:42:48 2022] GET /identity/v3/users/33ab6dea02224a0b981481767261f0ca => generated 293 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:48.904236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85390e0-7451-4cc3-9a77-45fb5d2771df None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:49.102219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015b846b-d459-42b4-b768-d638963f1753 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:49.194208 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85390e0-7451-4cc3-9a77-45fb5d2771df None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:49.207321 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:49.246905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a3b4ea-c7f5-4155-a7eb-bf87d740a7fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:49.250126 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a3b4ea-c7f5-4155-a7eb-bf87d740a7fd None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:49.251443 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a3b4ea-c7f5-4155-a7eb-bf87d740a7fd None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:49.254919 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a3b4ea-c7f5-4155-a7eb-bf87d740a7fd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:49.259203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a3b4ea-c7f5-4155-a7eb-bf87d740a7fd None admin] RBAC: Adding query filter params (group.id=6af07276afdc4cebab987faea681089d, role.id=1e04c9fea65244d28e1e59fc4c64f32f, scope.project.id=2e41e0644d514666af8ca7ed05ebaec8) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:49.259749 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a3b4ea-c7f5-4155-a7eb-bf87d740a7fd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:49.263330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a3b4ea-c7f5-4155-a7eb-bf87d740a7fd None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:49.275987 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 188/379] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Sep 18 00:42:48 2022] GET /identity/v3/role_assignments?group.id=6af07276afdc4cebab987faea681089d&scope.project.id=2e41e0644d514666af8ca7ed05ebaec8&role.id=1e04c9fea65244d28e1e59fc4c64f32f => generated 260 bytes in 405 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:49.278575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c257f37-2332-49e3-90fa-7a4230660027 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:49.279037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c257f37-2332-49e3-90fa-7a4230660027 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:49.279470 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c257f37-2332-49e3-90fa-7a4230660027 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:49.282161 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 189/380] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:49 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:49.292211 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a9542a-588c-40da-b5e4-e69da163ba04 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:49.293392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a9542a-588c-40da-b5e4-e69da163ba04 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:49.294140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a9542a-588c-40da-b5e4-e69da163ba04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:49.327353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-70a9542a-588c-40da-b5e4-e69da163ba04 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:49.329784 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-70a9542a-588c-40da-b5e4-e69da163ba04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:49.331719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6848f34-91de-42ad-817d-0e29066c290f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:49.332737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6848f34-91de-42ad-817d-0e29066c290f None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:49.333226 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6848f34-91de-42ad-817d-0e29066c290f None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:49.333664 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6848f34-91de-42ad-817d-0e29066c290f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:49.335270 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6848f34-91de-42ad-817d-0e29066c290f None admin] RBAC: Adding query filter params (role.id=a928285094b34996a609520c255b4c5d, user.id=33ab6dea02224a0b981481767261f0ca, scope.system=all) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:49.335823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6848f34-91de-42ad-817d-0e29066c290f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:49.338801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6848f34-91de-42ad-817d-0e29066c290f None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:49.377584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 192/381] 10.222.0.42 () {56 vars in 1308 bytes} [Sun Sep 18 00:42:48 2022] GET /identity/v3/role_assignments?user.id=33ab6dea02224a0b981481767261f0ca&scope.system=all&role.id=a928285094b34996a609520c255b4c5d => generated 510 bytes in 474 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:49.417396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:49.462193 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 190/382] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:42:49.966584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6848f34-91de-42ad-817d-0e29066c290f None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:50.227701 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6848f34-91de-42ad-817d-0e29066c290f None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:50.305316 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f67b25a-74ed-4265-8d89-b1c4d8f38f2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:50.306349 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f67b25a-74ed-4265-8d89-b1c4d8f38f2a None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:50.306862 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f67b25a-74ed-4265-8d89-b1c4d8f38f2a None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:50.307326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f67b25a-74ed-4265-8d89-b1c4d8f38f2a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:50.336294 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f67b25a-74ed-4265-8d89-b1c4d8f38f2a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:50.347677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-7f67b25a-74ed-4265-8d89-b1c4d8f38f2a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:50.347677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:50.347677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:50.347677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:50.347677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:50.347677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:50.347677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:50.347677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:50.347677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:50.347677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:50.347677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:50.347677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:50.347677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:50.347677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:50.347677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:50.347677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:50.349278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:50.349278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:50.349278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:50.349278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:50.349278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:50.349278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:50.349278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:50.349278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:50.349278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:50.349278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:50.349278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:50.349278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:50.349278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:50.349278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:50.349278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:50.349278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:50.350787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:50.350787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:50.350787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:50.350787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:50.350787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:50.350787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:50.350787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:50.350787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:50.350787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:50.350787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:50.350787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:50.350787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:42:50.350787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 193/383] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:49 2022] GET /identity/v3/roles/member => generated 84 bytes in 385 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:50.359715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a9542a-588c-40da-b5e4-e69da163ba04 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:50.589330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a9542a-588c-40da-b5e4-e69da163ba04 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:50.665296 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2225c455-4a3a-48b5-8b52-8d573773ad58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:50.666484 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2225c455-4a3a-48b5-8b52-8d573773ad58 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:50.667000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2225c455-4a3a-48b5-8b52-8d573773ad58 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:50.667492 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2225c455-4a3a-48b5-8b52-8d573773ad58 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:50.669359 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2225c455-4a3a-48b5-8b52-8d573773ad58 None admin] RBAC: Adding query filter params (name=member) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:50.669899 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2225c455-4a3a-48b5-8b52-8d573773ad58 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:50.672539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2225c455-4a3a-48b5-8b52-8d573773ad58 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:50.695464 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 191/384] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:50 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 337 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:50.704435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f67b25a-74ed-4265-8d89-b1c4d8f38f2a None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:50.769547 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:50.940678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f67b25a-74ed-4265-8d89-b1c4d8f38f2a None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:51.019700 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8149b42a-6b7a-4668-b0b1-1a72b9b45380 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:51.020793 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8149b42a-6b7a-4668-b0b1-1a72b9b45380 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:51.021249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8149b42a-6b7a-4668-b0b1-1a72b9b45380 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:51.021716 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8149b42a-6b7a-4668-b0b1-1a72b9b45380 None admin] PATH_INFO: `/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:51.023829 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8149b42a-6b7a-4668-b0b1-1a72b9b45380 None admin] RBAC: Authorizing `identity:get_group(group_id=cde72a6c511d455a8d0c64ae4d97ef26)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:51.026612 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8149b42a-6b7a-4668-b0b1-1a72b9b45380 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:51.030174 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 194/385] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 18 00:42:50 2022] GET /identity/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26 => generated 226 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:51.044471 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2225c455-4a3a-48b5-8b52-8d573773ad58 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:51.158863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:51.295442 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2225c455-4a3a-48b5-8b52-8d573773ad58 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:51.373075 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68215892-6901-495c-ba57-0efb19fbbe3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:51.374247 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68215892-6901-495c-ba57-0efb19fbbe3b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:51.374746 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68215892-6901-495c-ba57-0efb19fbbe3b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:51.375230 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68215892-6901-495c-ba57-0efb19fbbe3b None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:51.377277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68215892-6901-495c-ba57-0efb19fbbe3b None admin] RBAC: Authorizing `identity:get_project(project_id=4c46ff2118f948a085f639b37068aa57)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:51.380034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68215892-6901-495c-ba57-0efb19fbbe3b None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:51.383624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 192/386] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:51 2022] GET /identity/v3/projects/4c46ff2118f948a085f639b37068aa57 => generated 298 bytes in 341 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:51.392752 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8149b42a-6b7a-4668-b0b1-1a72b9b45380 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:51.584979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:51.608583 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8149b42a-6b7a-4668-b0b1-1a72b9b45380 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:51.686415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf2e450-672f-4404-ab2c-22203d47b133 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:51.687568 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf2e450-672f-4404-ab2c-22203d47b133 None admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:51.688050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf2e450-672f-4404-ab2c-22203d47b133 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:51.688525 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf2e450-672f-4404-ab2c-22203d47b133 None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57/groups/cde72a6c511d455a8d0c64ae4d97ef26/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:51.692083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf2e450-672f-4404-ab2c-22203d47b133 None admin] RBAC: Authorizing `identity:create_grant(project_id=4c46ff2118f948a085f639b37068aa57, group_id=cde72a6c511d455a8d0c64ae4d97ef26, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:51.694836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf2e450-672f-4404-ab2c-22203d47b133 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:51.726642 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 195/387] 10.222.0.42 () {58 vars in 1442 bytes} [Sun Sep 18 00:42:51 2022] PUT /identity/v3/projects/4c46ff2118f948a085f639b37068aa57/groups/cde72a6c511d455a8d0c64ae4d97ef26/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 329 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:42:52.349810 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16843cdb-8f4b-45ab-8a38-76ac5092fb96 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:52.350094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16843cdb-8f4b-45ab-8a38-76ac5092fb96 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:52.350519 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16843cdb-8f4b-45ab-8a38-76ac5092fb96 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:52.352127 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 193/388] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:52.363924 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3289a6d5-0b9e-4b5f-b979-2f9fcf4f585a None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:52.364676 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3289a6d5-0b9e-4b5f-b979-2f9fcf4f585a None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:52.364840 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3289a6d5-0b9e-4b5f-b979-2f9fcf4f585a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:52.401021 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-3289a6d5-0b9e-4b5f-b979-2f9fcf4f585a None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:52.404011 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-3289a6d5-0b9e-4b5f-b979-2f9fcf4f585a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:52.542412 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 196/389] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:52.987834 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3a85ca-e9fb-4bcc-b074-adf44a95d0dc None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:52.988952 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3a85ca-e9fb-4bcc-b074-adf44a95d0dc None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:52.989441 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3a85ca-e9fb-4bcc-b074-adf44a95d0dc None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:52.991365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 194/390] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:52.999547 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ef3807-a2f5-468d-972e-5f422464b34d None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:53.000131 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ef3807-a2f5-468d-972e-5f422464b34d None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:53.000648 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ef3807-a2f5-468d-972e-5f422464b34d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:53.023115 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-22ef3807-a2f5-468d-972e-5f422464b34d None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:53.024726 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-22ef3807-a2f5-468d-972e-5f422464b34d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:53.073248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3a85ca-e9fb-4bcc-b074-adf44a95d0dc None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:53.151389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 197/391] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:53.231114 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3a85ca-e9fb-4bcc-b074-adf44a95d0dc None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:53.242609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:53.272537 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b94530-0662-453c-80c6-5bead2d6d304 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:53.273194 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b94530-0662-453c-80c6-5bead2d6d304 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:53.273464 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b94530-0662-453c-80c6-5bead2d6d304 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:53.273900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b94530-0662-453c-80c6-5bead2d6d304 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:53.288005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b94530-0662-453c-80c6-5bead2d6d304 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:53.294189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-00b94530-0662-453c-80c6-5bead2d6d304 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:42:53.294189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:53.294189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:53.294189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:53.294189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:53.294189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:53.294189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:53.294189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:53.294189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:53.294189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:53.294189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:53.294189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:53.294189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:53.294189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:53.294189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:53.294189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:53.295286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:53.295286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:53.295286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:53.295286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:53.295286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:53.295286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:53.295286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:53.295286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:53.295286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:53.295286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:53.295286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:53.295286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:53.295286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:53.295286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:53.295286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:53.295286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:53.296184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:53.296184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:53.296184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:53.296184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:53.296184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:53.296184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:53.296184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:53.296184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:53.296184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:53.296184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:53.296184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:42:53.296184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:42:53.298634 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 195/392] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 18 00:42:53 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:53.307734 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ef3807-a2f5-468d-972e-5f422464b34d None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:53.339526 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:53.391528 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b384362-5a65-489f-a28a-5cf2a371e6f1 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:53.391784 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b384362-5a65-489f-a28a-5cf2a371e6f1 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:53.392069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b384362-5a65-489f-a28a-5cf2a371e6f1 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:53.393886 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 196/393] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:42:53.402566 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdd4566-c799-420b-8247-e181080f014c None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:53.402809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdd4566-c799-420b-8247-e181080f014c None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:53.403081 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdd4566-c799-420b-8247-e181080f014c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:53.434771 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-9fdd4566-c799-420b-8247-e181080f014c None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:53.438126 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-9fdd4566-c799-420b-8247-e181080f014c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:53.484955 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ef3807-a2f5-468d-972e-5f422464b34d None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:53.561269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2497617b-f5d6-4eac-a280-6df493742f9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:53.562913 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2497617b-f5d6-4eac-a280-6df493742f9d None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:53.563442 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2497617b-f5d6-4eac-a280-6df493742f9d None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:53.564122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2497617b-f5d6-4eac-a280-6df493742f9d None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:53.566266 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2497617b-f5d6-4eac-a280-6df493742f9d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:53.567243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2497617b-f5d6-4eac-a280-6df493742f9d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:53.569702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2497617b-f5d6-4eac-a280-6df493742f9d None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:53.588106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 198/395] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:42:53 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:53.588376 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 197/395] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:42:53.599919 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdd4566-c799-420b-8247-e181080f014c None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:53.729910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:53.734435 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdd4566-c799-420b-8247-e181080f014c None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:53.797055 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d3a27a-2fe9-4768-9300-ab2719133b46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:53.798435 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d3a27a-2fe9-4768-9300-ab2719133b46 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:53.799138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d3a27a-2fe9-4768-9300-ab2719133b46 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:53.800093 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d3a27a-2fe9-4768-9300-ab2719133b46 None admin] PATH_INFO: `/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:53.802591 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d3a27a-2fe9-4768-9300-ab2719133b46 None admin] RBAC: Authorizing `identity:get_group(group_id=cde72a6c511d455a8d0c64ae4d97ef26)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:53.805514 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d3a27a-2fe9-4768-9300-ab2719133b46 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:53.809685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 198/396] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 18 00:42:53 2022] GET /identity/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26 => generated 226 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:53.815713 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2497617b-f5d6-4eac-a280-6df493742f9d None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:53.831004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d3a27a-2fe9-4768-9300-ab2719133b46 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:54.023042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:54.050834 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2497617b-f5d6-4eac-a280-6df493742f9d None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:54.052630 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d3a27a-2fe9-4768-9300-ab2719133b46 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:54.088398 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcfad09-aaf1-437d-998e-64aaeb0b0c22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:54.089035 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcfad09-aaf1-437d-998e-64aaeb0b0c22 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:54.089314 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcfad09-aaf1-437d-998e-64aaeb0b0c22 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:54.089584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcfad09-aaf1-437d-998e-64aaeb0b0c22 None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:54.090919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dcfad09-aaf1-437d-998e-64aaeb0b0c22 None admin] RBAC: Authorizing `identity:get_project(project_id=4c46ff2118f948a085f639b37068aa57)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:54.092367 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dcfad09-aaf1-437d-998e-64aaeb0b0c22 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:54.094526 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 199/397] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:53 2022] GET /identity/v3/projects/4c46ff2118f948a085f639b37068aa57 => generated 298 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:54.095393 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcfad09-aaf1-437d-998e-64aaeb0b0c22 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:54.116416 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4fd0b0-456c-48ec-905d-f4866c8cb627 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:54.117711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4fd0b0-456c-48ec-905d-f4866c8cb627 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:54.118246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4fd0b0-456c-48ec-905d-f4866c8cb627 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:54.118835 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4fd0b0-456c-48ec-905d-f4866c8cb627 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:54.133142 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4fd0b0-456c-48ec-905d-f4866c8cb627 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:54.140582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-3f4fd0b0-456c-48ec-905d-f4866c8cb627 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:42:54.140582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:54.140582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:54.140582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:54.140582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:54.140582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:54.140582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:54.140582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:54.140582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:54.140582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:54.140582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:54.140582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:54.140582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:54.140582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:54.140582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:54.140582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:54.142022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:54.142022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:54.142022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:54.142022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:54.142022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:54.142022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:54.142022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:54.142022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:54.142022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:54.142022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:54.142022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:54.142022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:54.142022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:54.142022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:54.142022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:54.142022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:54.143327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:54.143327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:54.143327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:54.143327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:54.143327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:54.143327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:54.143327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:54.143327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:54.143327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:54.143327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:54.143327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:42:54.143327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:42:54.145855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 199/398] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 18 00:42:53 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 320 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:54.147729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4fd0b0-456c-48ec-905d-f4866c8cb627 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:54.188734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:54.216567 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcfad09-aaf1-437d-998e-64aaeb0b0c22 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:54.254407 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afb7c98-c357-44b3-a836-16af3333099c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:54.255068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afb7c98-c357-44b3-a836-16af3333099c None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:54.255342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afb7c98-c357-44b3-a836-16af3333099c None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:54.255609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afb7c98-c357-44b3-a836-16af3333099c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:54.264966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:54.271340 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afb7c98-c357-44b3-a836-16af3333099c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:54.276657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-1afb7c98-c357-44b3-a836-16af3333099c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:54.276657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:54.276657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:54.276657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:54.276657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:54.276657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:54.276657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:54.276657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:54.276657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:54.276657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:54.276657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:54.276657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:54.276657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:54.276657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:54.276657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:54.276657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:54.278306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:54.278306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:54.278306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:54.278306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:54.278306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:54.278306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:54.278306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:54.278306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:54.278306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:54.278306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:54.278306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:54.278306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:54.278306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:54.278306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:54.278306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:54.278306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:54.279917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:54.279917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:54.279917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:54.279917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:54.279917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:54.279917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:54.279917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:54.279917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:54.279917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:54.279917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:54.279917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:54.279917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:42:54.279917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 200/399] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:54 2022] GET /identity/v3/roles/member => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:54.279917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afb7c98-c357-44b3-a836-16af3333099c None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:54.329737 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4fd0b0-456c-48ec-905d-f4866c8cb627 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:54.393077 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc97d817-078d-46c6-bdaf-e9607fa9c790 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:54.394509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc97d817-078d-46c6-bdaf-e9607fa9c790 None admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:54.395205 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc97d817-078d-46c6-bdaf-e9607fa9c790 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:54.395756 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc97d817-078d-46c6-bdaf-e9607fa9c790 None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57/groups/cde72a6c511d455a8d0c64ae4d97ef26/roles/cf934cf252ce42d5a0445d3e156348d1` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:54.395913 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afb7c98-c357-44b3-a836-16af3333099c None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:54.402144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc97d817-078d-46c6-bdaf-e9607fa9c790 None admin] RBAC: Authorizing `identity:create_grant(project_id=4c46ff2118f948a085f639b37068aa57, group_id=cde72a6c511d455a8d0c64ae4d97ef26, role_id=cf934cf252ce42d5a0445d3e156348d1)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:54.404868 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc97d817-078d-46c6-bdaf-e9607fa9c790 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:54.435225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 200/400] 10.222.0.42 () {58 vars in 1442 bytes} [Sun Sep 18 00:42:54 2022] PUT /identity/v3/projects/4c46ff2118f948a085f639b37068aa57/groups/cde72a6c511d455a8d0c64ae4d97ef26/roles/cf934cf252ce42d5a0445d3e156348d1 => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:42:54.437239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc97d817-078d-46c6-bdaf-e9607fa9c790 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:54.438939 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfa400d-d20c-4c03-93da-38ca3baa2ee2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:54.439570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfa400d-d20c-4c03-93da-38ca3baa2ee2 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:54.439846 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfa400d-d20c-4c03-93da-38ca3baa2ee2 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:54.440120 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfa400d-d20c-4c03-93da-38ca3baa2ee2 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:54.440812 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cfa400d-d20c-4c03-93da-38ca3baa2ee2 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:54.441118 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cfa400d-d20c-4c03-93da-38ca3baa2ee2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:54.442639 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cfa400d-d20c-4c03-93da-38ca3baa2ee2 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:54.459400 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 201/401] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:42:54 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:54.461391 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfa400d-d20c-4c03-93da-38ca3baa2ee2 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:54.584313 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:54.640073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfa400d-d20c-4c03-93da-38ca3baa2ee2 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:54.685959 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717aec2d-44ed-4dfa-b8ab-bae750abcf9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:54.686645 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717aec2d-44ed-4dfa-b8ab-bae750abcf9b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:54.686922 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717aec2d-44ed-4dfa-b8ab-bae750abcf9b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:54.687185 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717aec2d-44ed-4dfa-b8ab-bae750abcf9b None admin] PATH_INFO: `/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:54.689207 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-717aec2d-44ed-4dfa-b8ab-bae750abcf9b None admin] RBAC: Authorizing `identity:get_group(group_id=cde72a6c511d455a8d0c64ae4d97ef26)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:54.690845 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-717aec2d-44ed-4dfa-b8ab-bae750abcf9b None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:54.695996 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:54.697866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 202/402] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 18 00:42:54 2022] GET /identity/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26 => generated 226 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:54.699468 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717aec2d-44ed-4dfa-b8ab-bae750abcf9b None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:54.726381 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc97d817-078d-46c6-bdaf-e9607fa9c790 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:54.795995 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3f08c1-6c39-4076-aef0-8602e7186cbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:54.797446 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3f08c1-6c39-4076-aef0-8602e7186cbe None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:54.797955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3f08c1-6c39-4076-aef0-8602e7186cbe None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:54.798823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3f08c1-6c39-4076-aef0-8602e7186cbe None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:54.801353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3f08c1-6c39-4076-aef0-8602e7186cbe None admin] RBAC: Adding query filter params (name=member) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:54.802230 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3f08c1-6c39-4076-aef0-8602e7186cbe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:54.806610 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3f08c1-6c39-4076-aef0-8602e7186cbe None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:54.822793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 201/403] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:54 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 386 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:54.824775 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db7b98f-549a-41e4-b367-d445a1ffb363 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:54.825069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db7b98f-549a-41e4-b367-d445a1ffb363 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:54.825340 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db7b98f-549a-41e4-b367-d445a1ffb363 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:54.826932 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 202/404] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:54.831478 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db7b98f-549a-41e4-b367-d445a1ffb363 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:54.871517 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717aec2d-44ed-4dfa-b8ab-bae750abcf9b None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:54.916259 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:54.924557 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57878188-4699-4b3b-a735-3d088494fa67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:54.925450 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57878188-4699-4b3b-a735-3d088494fa67 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:54.925825 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57878188-4699-4b3b-a735-3d088494fa67 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:54.926036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57878188-4699-4b3b-a735-3d088494fa67 None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:54.927233 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57878188-4699-4b3b-a735-3d088494fa67 None admin] RBAC: Authorizing `identity:get_project(project_id=8085b2496c6847aa83e809bde1f3e986)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:54.928678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57878188-4699-4b3b-a735-3d088494fa67 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:54.930354 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 203/405] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:54 2022] GET /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986 => generated 302 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:54.931546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af223e5e-90d5-4f4a-baf6-a898b28dd999 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:54.931907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af223e5e-90d5-4f4a-baf6-a898b28dd999 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:54.932189 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af223e5e-90d5-4f4a-baf6-a898b28dd999 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:54.953091 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-af223e5e-90d5-4f4a-baf6-a898b28dd999 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:54.954438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-af223e5e-90d5-4f4a-baf6-a898b28dd999 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:54.987288 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db7b98f-549a-41e4-b367-d445a1ffb363 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:55.029873 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d17ddd-a370-4fd9-b2cd-8d89875f8b6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:55.030604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d17ddd-a370-4fd9-b2cd-8d89875f8b6f None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:55.030892 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d17ddd-a370-4fd9-b2cd-8d89875f8b6f None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:55.031168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d17ddd-a370-4fd9-b2cd-8d89875f8b6f None admin] PATH_INFO: `/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:55.032312 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d17ddd-a370-4fd9-b2cd-8d89875f8b6f None admin] RBAC: Authorizing `identity:get_group(group_id=cde72a6c511d455a8d0c64ae4d97ef26)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:55.033825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d17ddd-a370-4fd9-b2cd-8d89875f8b6f None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:55.035521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 203/406] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 18 00:42:54 2022] GET /identity/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26 => generated 226 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:55.036795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c732bdcf-8da4-426c-aee1-73f2e649b962 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:55.037159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c732bdcf-8da4-426c-aee1-73f2e649b962 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:55.037433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c732bdcf-8da4-426c-aee1-73f2e649b962 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:55.043133 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 204/407] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:55.043133 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c732bdcf-8da4-426c-aee1-73f2e649b962 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:55.074705 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 204/408] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:55.074705 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af223e5e-90d5-4f4a-baf6-a898b28dd999 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:55.079160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:55.206209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c732bdcf-8da4-426c-aee1-73f2e649b962 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:55.259303 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e086b6f4-9fbf-4f8e-a45b-650fed525520 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:55.260113 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e086b6f4-9fbf-4f8e-a45b-650fed525520 None admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:55.260469 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e086b6f4-9fbf-4f8e-a45b-650fed525520 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:55.260821 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e086b6f4-9fbf-4f8e-a45b-650fed525520 None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986/groups/cde72a6c511d455a8d0c64ae4d97ef26/roles/cf934cf252ce42d5a0445d3e156348d1` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:55.262427 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af223e5e-90d5-4f4a-baf6-a898b28dd999 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:55.263477 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e086b6f4-9fbf-4f8e-a45b-650fed525520 None admin] RBAC: Authorizing `identity:create_grant(project_id=8085b2496c6847aa83e809bde1f3e986, group_id=cde72a6c511d455a8d0c64ae4d97ef26, role_id=cf934cf252ce42d5a0445d3e156348d1)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:55.265336 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e086b6f4-9fbf-4f8e-a45b-650fed525520 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:55.286548 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 205/409] 10.222.0.42 () {58 vars in 1442 bytes} [Sun Sep 18 00:42:55 2022] PUT /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986/groups/cde72a6c511d455a8d0c64ae4d97ef26/roles/cf934cf252ce42d5a0445d3e156348d1 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:42:55.288330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9ab8cc-69a2-4d4f-8238-5c705f0b7b1b None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:55.289019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9ab8cc-69a2-4d4f-8238-5c705f0b7b1b None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:55.289396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9ab8cc-69a2-4d4f-8238-5c705f0b7b1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:55.331339 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-3a9ab8cc-69a2-4d4f-8238-5c705f0b7b1b None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:55.333579 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3a9ab8cc-69a2-4d4f-8238-5c705f0b7b1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:55.357122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:55.388029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792ad8d8-5a7a-46fe-8a05-0c9e8671d1c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:55.388726 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792ad8d8-5a7a-46fe-8a05-0c9e8671d1c8 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:55.389005 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792ad8d8-5a7a-46fe-8a05-0c9e8671d1c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:55.389274 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792ad8d8-5a7a-46fe-8a05-0c9e8671d1c8 None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:55.390624 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792ad8d8-5a7a-46fe-8a05-0c9e8671d1c8 None admin] RBAC: Authorizing `identity:get_project(project_id=8085b2496c6847aa83e809bde1f3e986)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:55.392144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792ad8d8-5a7a-46fe-8a05-0c9e8671d1c8 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:55.394201 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 205/410] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:55 2022] GET /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986 => generated 302 bytes in 330 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:55.395758 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792ad8d8-5a7a-46fe-8a05-0c9e8671d1c8 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:55.429049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 206/411] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:55.430646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9ab8cc-69a2-4d4f-8238-5c705f0b7b1b None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:55.446679 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:55.567654 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792ad8d8-5a7a-46fe-8a05-0c9e8671d1c8 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:55.615855 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c73bc9f-6128-4eae-9bc1-bb91462bdf3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:55.616537 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c73bc9f-6128-4eae-9bc1-bb91462bdf3f None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:55.616840 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c73bc9f-6128-4eae-9bc1-bb91462bdf3f None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:55.617077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c73bc9f-6128-4eae-9bc1-bb91462bdf3f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:55.623502 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9ab8cc-69a2-4d4f-8238-5c705f0b7b1b None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:55.655878 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c73bc9f-6128-4eae-9bc1-bb91462bdf3f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:55.673053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-9c73bc9f-6128-4eae-9bc1-bb91462bdf3f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:42:55.673053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:55.673053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:55.673053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:55.673053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:55.673053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:55.673053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:55.673053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:55.673053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:55.673053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:55.673053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:55.673053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:55.673053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:55.673053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:55.673053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:55.673053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:55.679221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:55.679221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:55.679221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:55.679221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:55.679221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:55.679221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:55.679221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:55.679221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:55.679221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:55.679221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:55.679221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:55.679221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:55.679221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:55.679221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:55.679221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:55.679221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:55.680744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:55.680744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:55.680744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:55.680744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:55.680744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:55.680744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:55.680744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:55.680744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:55.680744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:55.680744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:55.680744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:42:55.680744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:42:55.680744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 206/412] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:42:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:42:55.690609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c73bc9f-6128-4eae-9bc1-bb91462bdf3f None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:55.755280 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b40157-32d2-4fa6-bd11-f000c73a2976 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:55.756467 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b40157-32d2-4fa6-bd11-f000c73a2976 None admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:55.756467 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b40157-32d2-4fa6-bd11-f000c73a2976 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:55.756665 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b40157-32d2-4fa6-bd11-f000c73a2976 None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986/groups/cde72a6c511d455a8d0c64ae4d97ef26/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:55.766943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b40157-32d2-4fa6-bd11-f000c73a2976 None admin] RBAC: Authorizing `identity:create_grant(project_id=8085b2496c6847aa83e809bde1f3e986, group_id=cde72a6c511d455a8d0c64ae4d97ef26, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:55.768447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b40157-32d2-4fa6-bd11-f000c73a2976 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:55.779902 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:55.798705 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 207/413] 10.222.0.42 () {58 vars in 1442 bytes} [Sun Sep 18 00:42:55 2022] PUT /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986/groups/cde72a6c511d455a8d0c64ae4d97ef26/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:42:55.827487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c73bc9f-6128-4eae-9bc1-bb91462bdf3f None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:55.848418 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b40157-32d2-4fa6-bd11-f000c73a2976 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:55.884124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d6b5d6-7383-4bc2-a997-82c3d82e75d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:55.885195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d6b5d6-7383-4bc2-a997-82c3d82e75d9 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:55.885673 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d6b5d6-7383-4bc2-a997-82c3d82e75d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:55.886168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d6b5d6-7383-4bc2-a997-82c3d82e75d9 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:55.887909 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d6b5d6-7383-4bc2-a997-82c3d82e75d9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:55.888470 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d6b5d6-7383-4bc2-a997-82c3d82e75d9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:55.890298 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d6b5d6-7383-4bc2-a997-82c3d82e75d9 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:55.901274 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 207/414] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:55.907889 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d6b5d6-7383-4bc2-a997-82c3d82e75d9 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:56.050870 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d6b5d6-7383-4bc2-a997-82c3d82e75d9 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:56.061364 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:56.100207 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:56.144515 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b40157-32d2-4fa6-bd11-f000c73a2976 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:56.147807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64217979-bb93-42bf-80bf-b7c68be9e566 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:56.148477 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64217979-bb93-42bf-80bf-b7c68be9e566 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:56.148792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64217979-bb93-42bf-80bf-b7c68be9e566 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:56.149038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64217979-bb93-42bf-80bf-b7c68be9e566 None admin] PATH_INFO: `/v3/groups/6af07276afdc4cebab987faea681089d` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:56.152102 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64217979-bb93-42bf-80bf-b7c68be9e566 None admin] RBAC: Authorizing `identity:get_group(group_id=6af07276afdc4cebab987faea681089d)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:56.154139 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64217979-bb93-42bf-80bf-b7c68be9e566 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:56.155915 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 208/415] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 18 00:42:55 2022] GET /identity/v3/groups/6af07276afdc4cebab987faea681089d => generated 229 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:56.164212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64217979-bb93-42bf-80bf-b7c68be9e566 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:56.217230 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a817b4cd-3e12-4c38-9fb9-c2eb94f54f96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:56.217889 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a817b4cd-3e12-4c38-9fb9-c2eb94f54f96 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:56.218171 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a817b4cd-3e12-4c38-9fb9-c2eb94f54f96 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:56.218446 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a817b4cd-3e12-4c38-9fb9-c2eb94f54f96 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:56.241996 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a817b4cd-3e12-4c38-9fb9-c2eb94f54f96 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:56.247714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-a817b4cd-3e12-4c38-9fb9-c2eb94f54f96 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:56.247714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:56.247714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:56.247714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:56.247714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:56.247714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:56.247714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:56.247714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:56.247714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:56.247714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:56.247714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:56.247714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:56.247714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:56.247714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:56.247714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:56.247714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:56.249239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:56.249239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:56.249239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:56.249239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:56.249239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:56.249239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:56.249239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:56.249239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:56.249239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:56.249239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:56.249239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:56.249239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:56.249239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:56.249239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:56.249239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:56.249239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:56.250154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:56.250154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:56.250154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:56.250154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:56.250154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:56.250154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:56.250154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:56.250154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:56.250154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:56.250154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:56.250154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:56.250154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:42:56.250154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 208/416] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:55 2022] GET /identity/v3/roles/member => generated 84 bytes in 402 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:56.260128 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a817b4cd-3e12-4c38-9fb9-c2eb94f54f96 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:56.292405 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64217979-bb93-42bf-80bf-b7c68be9e566 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:56.330192 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd71a76e-d968-4957-91f7-7786fe986cda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:56.330816 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd71a76e-d968-4957-91f7-7786fe986cda None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:56.331099 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd71a76e-d968-4957-91f7-7786fe986cda None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:56.331374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd71a76e-d968-4957-91f7-7786fe986cda None admin] PATH_INFO: `/v3/projects/2e41e0644d514666af8ca7ed05ebaec8` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:56.332606 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd71a76e-d968-4957-91f7-7786fe986cda None admin] RBAC: Authorizing `identity:get_project(project_id=2e41e0644d514666af8ca7ed05ebaec8)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:56.334065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd71a76e-d968-4957-91f7-7786fe986cda None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:56.336126 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 209/417] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:56 2022] GET /identity/v3/projects/2e41e0644d514666af8ca7ed05ebaec8 => generated 344 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:56.343235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd71a76e-d968-4957-91f7-7786fe986cda None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:56.465395 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:56.489575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a817b4cd-3e12-4c38-9fb9-c2eb94f54f96 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:56.524747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd71a76e-d968-4957-91f7-7786fe986cda None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:56.561075 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1479f65-865c-4eb2-a71a-2cf6be04157d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:56.562026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1479f65-865c-4eb2-a71a-2cf6be04157d None admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:56.562406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1479f65-865c-4eb2-a71a-2cf6be04157d None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:56.562814 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1479f65-865c-4eb2-a71a-2cf6be04157d None admin] PATH_INFO: `/v3/projects/2e41e0644d514666af8ca7ed05ebaec8/groups/6af07276afdc4cebab987faea681089d/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:56.564944 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1479f65-865c-4eb2-a71a-2cf6be04157d None admin] RBAC: Authorizing `identity:create_grant(project_id=2e41e0644d514666af8ca7ed05ebaec8, group_id=6af07276afdc4cebab987faea681089d, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:56.566015 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:56.566817 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1479f65-865c-4eb2-a71a-2cf6be04157d None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:56.589151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 210/418] 10.222.0.42 () {58 vars in 1442 bytes} [Sun Sep 18 00:42:56 2022] PUT /identity/v3/projects/2e41e0644d514666af8ca7ed05ebaec8/groups/6af07276afdc4cebab987faea681089d/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:42:56.589919 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8540e226-2180-4b47-a79a-b5b08bcea39a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:56.590727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8540e226-2180-4b47-a79a-b5b08bcea39a None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:56.591187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8540e226-2180-4b47-a79a-b5b08bcea39a None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:56.591598 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8540e226-2180-4b47-a79a-b5b08bcea39a None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:56.592470 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8540e226-2180-4b47-a79a-b5b08bcea39a None admin] RBAC: Adding query filter params (name=member) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:56.592916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8540e226-2180-4b47-a79a-b5b08bcea39a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:56.594616 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8540e226-2180-4b47-a79a-b5b08bcea39a None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:56.607544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 209/419] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:56 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 348 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:56.615558 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1479f65-865c-4eb2-a71a-2cf6be04157d None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:56.797464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1479f65-865c-4eb2-a71a-2cf6be04157d None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:56.818573 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:56.845474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046e92d7-3fd5-4dff-87d3-587ffdffcd7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:56.846018 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046e92d7-3fd5-4dff-87d3-587ffdffcd7d None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:56.846296 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046e92d7-3fd5-4dff-87d3-587ffdffcd7d None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:56.846620 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046e92d7-3fd5-4dff-87d3-587ffdffcd7d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:56.934782 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046e92d7-3fd5-4dff-87d3-587ffdffcd7d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:56.936591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046e92d7-3fd5-4dff-87d3-587ffdffcd7d None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:57.066811 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 211/420] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:42:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 448 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:42:57.068611 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8540e226-2180-4b47-a79a-b5b08bcea39a None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:57.224065 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8540e226-2180-4b47-a79a-b5b08bcea39a None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:57.275080 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f272eb-d5fa-4676-b223-f72380d3e729 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:57.276569 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f272eb-d5fa-4676-b223-f72380d3e729 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:57.277346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f272eb-d5fa-4676-b223-f72380d3e729 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:57.278109 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f272eb-d5fa-4676-b223-f72380d3e729 None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:57.283530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f272eb-d5fa-4676-b223-f72380d3e729 None admin] RBAC: Authorizing `identity:get_project(project_id=4c46ff2118f948a085f639b37068aa57)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:57.285603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f272eb-d5fa-4676-b223-f72380d3e729 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:57.288379 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 210/421] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:57 2022] GET /identity/v3/projects/4c46ff2118f948a085f639b37068aa57 => generated 298 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:57.294851 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046e92d7-3fd5-4dff-87d3-587ffdffcd7d None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:57.325433 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:57.390660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1960526a-3c6e-42d1-a460-a92f22a3fe51 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:57.391301 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1960526a-3c6e-42d1-a460-a92f22a3fe51 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:57.391776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1960526a-3c6e-42d1-a460-a92f22a3fe51 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:57.393334 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 211/422] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:57.398972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac15aad-f8b5-4ba7-860a-f847fe52e660 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:57.399551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac15aad-f8b5-4ba7-860a-f847fe52e660 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:57.400049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac15aad-f8b5-4ba7-860a-f847fe52e660 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:57.430562 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046e92d7-3fd5-4dff-87d3-587ffdffcd7d None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:57.438166 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-1ac15aad-f8b5-4ba7-860a-f847fe52e660 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:57.439763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-1ac15aad-f8b5-4ba7-860a-f847fe52e660 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:57.474529 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cecade2-3403-4b60-872f-814c3d576ce4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:57.475100 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cecade2-3403-4b60-872f-814c3d576ce4 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:57.475378 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cecade2-3403-4b60-872f-814c3d576ce4 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:57.475647 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cecade2-3403-4b60-872f-814c3d576ce4 None admin] PATH_INFO: `/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:57.476806 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cecade2-3403-4b60-872f-814c3d576ce4 None admin] RBAC: Authorizing `identity:get_group(group_id=cde72a6c511d455a8d0c64ae4d97ef26)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:57.478371 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cecade2-3403-4b60-872f-814c3d576ce4 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:57.480561 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 212/423] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 18 00:42:57 2022] GET /identity/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26 => generated 226 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:57.486472 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cecade2-3403-4b60-872f-814c3d576ce4 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:57.511622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:57.536843 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 212/424] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:42:57.626800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:57.648766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cecade2-3403-4b60-872f-814c3d576ce4 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:57.698143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da21d8cb-53d3-4040-ba14-23c34ca7570e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:57.698820 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da21d8cb-53d3-4040-ba14-23c34ca7570e None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:57.699081 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da21d8cb-53d3-4040-ba14-23c34ca7570e None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:57.699427 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da21d8cb-53d3-4040-ba14-23c34ca7570e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:57.700168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da21d8cb-53d3-4040-ba14-23c34ca7570e None admin] RBAC: Adding query filter params (group.id=cde72a6c511d455a8d0c64ae4d97ef26, role.id=18f6cee478e54567b70d5c7557fe694e, scope.project.id=4c46ff2118f948a085f639b37068aa57) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:57.700480 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da21d8cb-53d3-4040-ba14-23c34ca7570e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:57.702000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da21d8cb-53d3-4040-ba14-23c34ca7570e None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:57.711550 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 213/425] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Sep 18 00:42:57 2022] GET /identity/v3/role_assignments?group.id=cde72a6c511d455a8d0c64ae4d97ef26&scope.project.id=4c46ff2118f948a085f639b37068aa57&role.id=18f6cee478e54567b70d5c7557fe694e => generated 611 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:42:57.855940 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac15aad-f8b5-4ba7-860a-f847fe52e660 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:57.978809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac15aad-f8b5-4ba7-860a-f847fe52e660 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:58.030596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700e8b59-8d40-430b-9e62-885136d7f043 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:58.032106 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700e8b59-8d40-430b-9e62-885136d7f043 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:58.032906 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700e8b59-8d40-430b-9e62-885136d7f043 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:58.033522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700e8b59-8d40-430b-9e62-885136d7f043 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:58.049280 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700e8b59-8d40-430b-9e62-885136d7f043 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:58.056617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-700e8b59-8d40-430b-9e62-885136d7f043 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:42:58.056617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:58.056617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:58.056617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:58.056617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:58.056617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:58.056617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:58.056617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:58.056617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:58.056617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:58.056617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:58.056617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:58.056617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:58.056617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:58.056617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:58.056617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:58.057914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:58.057914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:58.057914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:58.057914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:58.057914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:58.057914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:58.057914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:58.057914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:58.057914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:58.057914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:58.057914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:58.057914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:58.057914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:58.057914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:58.057914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:58.057914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:58.059979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:58.059979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:58.059979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:58.059979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:58.059979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:58.059979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:58.059979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:58.059979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:58.059979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:58.059979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:58.059979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:42:58.059979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:42:58.063717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 213/426] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 18 00:42:57 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:58.073414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da21d8cb-53d3-4040-ba14-23c34ca7570e None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:58.266499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da21d8cb-53d3-4040-ba14-23c34ca7570e None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:58.330559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b912840c-b77c-4462-a273-6d1c364632a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:58.332058 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b912840c-b77c-4462-a273-6d1c364632a2 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:58.332838 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b912840c-b77c-4462-a273-6d1c364632a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:58.333604 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b912840c-b77c-4462-a273-6d1c364632a2 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:58.335158 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b912840c-b77c-4462-a273-6d1c364632a2 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:58.336000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b912840c-b77c-4462-a273-6d1c364632a2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:58.339138 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b912840c-b77c-4462-a273-6d1c364632a2 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:58.355810 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 214/427] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:42:58 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:58.364922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700e8b59-8d40-430b-9e62-885136d7f043 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:58.469353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:58.491323 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32459e05-8dd7-456e-945f-36819b0aa4e6 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:58.492636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32459e05-8dd7-456e-945f-36819b0aa4e6 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:58.493747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32459e05-8dd7-456e-945f-36819b0aa4e6 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:58.497635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 215/428] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:58 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:58.508160 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196f02cc-de9b-46a0-953a-76ecaaa2e22f None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:58.508556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196f02cc-de9b-46a0-953a-76ecaaa2e22f None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:58.508726 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196f02cc-de9b-46a0-953a-76ecaaa2e22f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:58.544965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700e8b59-8d40-430b-9e62-885136d7f043 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:58.559277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-196f02cc-de9b-46a0-953a-76ecaaa2e22f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:58.561145 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-196f02cc-de9b-46a0-953a-76ecaaa2e22f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:58.563390 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:58.616233 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6068407a-acba-43d7-a380-ab1dd88a65a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:58.617750 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6068407a-acba-43d7-a380-ab1dd88a65a4 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:58.618246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6068407a-acba-43d7-a380-ab1dd88a65a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:58.618821 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6068407a-acba-43d7-a380-ab1dd88a65a4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:58.657738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 216/429] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:58.659254 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823d16ce-ade4-4918-9ff9-d970e0f00018 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:58.659636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823d16ce-ade4-4918-9ff9-d970e0f00018 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:58.659919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823d16ce-ade4-4918-9ff9-d970e0f00018 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:58.661488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 217/430] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:58.668748 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9242ee81-5eb9-46f2-8840-24a78189e67d None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:58.669198 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9242ee81-5eb9-46f2-8840-24a78189e67d None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:58.669576 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9242ee81-5eb9-46f2-8840-24a78189e67d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:58.692634 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-9242ee81-5eb9-46f2-8840-24a78189e67d None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:58.694753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-9242ee81-5eb9-46f2-8840-24a78189e67d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:58.733266 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6068407a-acba-43d7-a380-ab1dd88a65a4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:58.735568 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6068407a-acba-43d7-a380-ab1dd88a65a4 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:58.778349 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 218/431] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:58.823061 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:58.858916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 214/432] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:42:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 495 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:42:58.864962 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9242ee81-5eb9-46f2-8840-24a78189e67d None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:58.961265 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:58.994722 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9242ee81-5eb9-46f2-8840-24a78189e67d None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:59.032480 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96144b49-bafa-48aa-9327-2e59c79bd05a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:59.033047 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96144b49-bafa-48aa-9327-2e59c79bd05a None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:59.033326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96144b49-bafa-48aa-9327-2e59c79bd05a None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:59.033599 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96144b49-bafa-48aa-9327-2e59c79bd05a None admin] PATH_INFO: `/v3/projects/4c46ff2118f948a085f639b37068aa57` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:59.034783 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96144b49-bafa-48aa-9327-2e59c79bd05a None admin] RBAC: Authorizing `identity:get_project(project_id=4c46ff2118f948a085f639b37068aa57)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:59.036220 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96144b49-bafa-48aa-9327-2e59c79bd05a None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:59.042523 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 219/433] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:42:58 2022] GET /identity/v3/projects/4c46ff2118f948a085f639b37068aa57 => generated 298 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:59.044119 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6068407a-acba-43d7-a380-ab1dd88a65a4 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:59.098322 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96144b49-bafa-48aa-9327-2e59c79bd05a None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:59.160521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6068407a-acba-43d7-a380-ab1dd88a65a4 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:59.201193 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27093c4-2094-4132-b13c-354d8b743853 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:59.202199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27093c4-2094-4132-b13c-354d8b743853 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:59.202743 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27093c4-2094-4132-b13c-354d8b743853 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:59.203213 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27093c4-2094-4132-b13c-354d8b743853 None admin] PATH_INFO: `/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:59.204566 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27093c4-2094-4132-b13c-354d8b743853 None admin] RBAC: Authorizing `identity:get_group(group_id=cde72a6c511d455a8d0c64ae4d97ef26)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:59.206259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27093c4-2094-4132-b13c-354d8b743853 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:59.213310 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 215/434] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 18 00:42:59 2022] GET /identity/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26 => generated 226 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:59.214652 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27093c4-2094-4132-b13c-354d8b743853 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:59.232659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:59.237472 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96144b49-bafa-48aa-9327-2e59c79bd05a None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:59.275303 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001adfe5-2bc1-49d4-b180-fada0a6d783a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:59.276205 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001adfe5-2bc1-49d4-b180-fada0a6d783a None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:59.276658 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001adfe5-2bc1-49d4-b180-fada0a6d783a None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:59.277075 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001adfe5-2bc1-49d4-b180-fada0a6d783a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:59.280865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:59.293647 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001adfe5-2bc1-49d4-b180-fada0a6d783a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:59.299199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-001adfe5-2bc1-49d4-b180-fada0a6d783a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:59.299199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:59.299199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:59.299199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:59.299199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:59.299199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:59.299199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:59.299199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:59.299199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:59.299199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:59.299199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:59.299199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:59.299199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:59.299199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:59.299199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:59.299199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:59.300438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:59.300438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:59.300438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:59.300438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:59.300438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:59.300438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:59.300438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:59.300438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:59.300438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:59.300438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:59.300438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:59.300438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:59.300438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:59.300438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:59.300438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:59.300438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:59.301359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:59.301359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:59.301359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:59.301359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:59.301359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:59.301359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:59.301359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:59.301359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:59.301359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:59.301359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:59.301359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 18 00:42:59.301359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:42:59.301359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 220/435] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:42:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:42:59.302048 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001adfe5-2bc1-49d4-b180-fada0a6d783a None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:59.357278 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27093c4-2094-4132-b13c-354d8b743853 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:59.401130 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897956c2-515c-4d83-b839-4d5a7a674755 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:59.403331 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897956c2-515c-4d83-b839-4d5a7a674755 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:59.404048 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897956c2-515c-4d83-b839-4d5a7a674755 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:59.404731 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897956c2-515c-4d83-b839-4d5a7a674755 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:59.406111 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897956c2-515c-4d83-b839-4d5a7a674755 None admin] RBAC: Adding query filter params (group.id=cde72a6c511d455a8d0c64ae4d97ef26, role.id=cf934cf252ce42d5a0445d3e156348d1, scope.project.id=4c46ff2118f948a085f639b37068aa57) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:59.406637 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897956c2-515c-4d83-b839-4d5a7a674755 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:59.409267 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897956c2-515c-4d83-b839-4d5a7a674755 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:59.419649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 216/436] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Sep 18 00:42:59 2022] GET /identity/v3/role_assignments?group.id=cde72a6c511d455a8d0c64ae4d97ef26&scope.project.id=4c46ff2118f948a085f639b37068aa57&role.id=cf934cf252ce42d5a0445d3e156348d1 => generated 611 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:59.421097 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897956c2-515c-4d83-b839-4d5a7a674755 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:59.432157 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001adfe5-2bc1-49d4-b180-fada0a6d783a None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:59.483899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3d0412-2c45-403b-983e-3b05c8849bd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:59.485059 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3d0412-2c45-403b-983e-3b05c8849bd7 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:59.485576 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3d0412-2c45-403b-983e-3b05c8849bd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:59.486080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3d0412-2c45-403b-983e-3b05c8849bd7 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:59.506541 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3d0412-2c45-403b-983e-3b05c8849bd7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:42:59.517104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-dc3d0412-2c45-403b-983e-3b05c8849bd7 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:42:59.517104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:42:59.517104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:42:59.517104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:42:59.517104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:42:59.517104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:42:59.517104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:42:59.517104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:42:59.517104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:42:59.517104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:42:59.517104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:42:59.517104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:42:59.517104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:42:59.517104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:42:59.517104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:42:59.517104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:42:59.518090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:42:59.518090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:42:59.518090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:42:59.518090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:42:59.518090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:42:59.518090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:42:59.518090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:42:59.518090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:42:59.518090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:42:59.518090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:42:59.518090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:42:59.518090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:42:59.518090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:42:59.518090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:42:59.518090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:42:59.518090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:42:59.518979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:42:59.518979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:42:59.518979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:42:59.518979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:42:59.518979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:42:59.518979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:42:59.518979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:42:59.518979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:42:59.518979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:42:59.518979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:42:59.518979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:42:59.518979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:42:59.528458 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 221/437] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 18 00:42:59 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:42:59.530971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6b11c5-0c37-47b3-aa5c-5944a770cefc None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:59.531513 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6b11c5-0c37-47b3-aa5c-5944a770cefc None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:59.532120 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6b11c5-0c37-47b3-aa5c-5944a770cefc None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:59.535799 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 222/438] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:42:59 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:42:59.537004 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6b11c5-0c37-47b3-aa5c-5944a770cefc None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:59.546958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:59.583006 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897956c2-515c-4d83-b839-4d5a7a674755 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:59.614671 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:59.640455 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f590e8-0506-4d12-9ebb-5a400c6caa49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:59.641092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f590e8-0506-4d12-9ebb-5a400c6caa49 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:59.641374 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f590e8-0506-4d12-9ebb-5a400c6caa49 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:59.641646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f590e8-0506-4d12-9ebb-5a400c6caa49 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:59.642337 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f590e8-0506-4d12-9ebb-5a400c6caa49 None admin] RBAC: Adding query filter params (name=member) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:59.642668 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f590e8-0506-4d12-9ebb-5a400c6caa49 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:59.644049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f590e8-0506-4d12-9ebb-5a400c6caa49 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:59.655119 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 217/439] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:42:59 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:59.656374 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2297d2c8-adac-42b4-966f-7dcf7504d1f2 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:59.656658 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2297d2c8-adac-42b4-966f-7dcf7504d1f2 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:59.656933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2297d2c8-adac-42b4-966f-7dcf7504d1f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:59.678200 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-2297d2c8-adac-42b4-966f-7dcf7504d1f2 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:42:59.679460 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-2297d2c8-adac-42b4-966f-7dcf7504d1f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:42:59.687625 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6b11c5-0c37-47b3-aa5c-5944a770cefc None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:59.729895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1bf7f4-d562-41da-b867-7b9338e9c07d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:59.730651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1bf7f4-d562-41da-b867-7b9338e9c07d None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:59.731826 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1bf7f4-d562-41da-b867-7b9338e9c07d None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:59.732120 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1bf7f4-d562-41da-b867-7b9338e9c07d None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:59.732874 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1bf7f4-d562-41da-b867-7b9338e9c07d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:42:59.733118 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1bf7f4-d562-41da-b867-7b9338e9c07d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:42:59.734540 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1bf7f4-d562-41da-b867-7b9338e9c07d None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:42:59.744551 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 223/440] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:42:59 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:42:59.745552 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1bf7f4-d562-41da-b867-7b9338e9c07d None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:59.781876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 218/441] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:42:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:42:59.784177 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2297d2c8-adac-42b4-966f-7dcf7504d1f2 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:42:59.886662 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:59.916692 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1bf7f4-d562-41da-b867-7b9338e9c07d None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:59.944796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2297d2c8-adac-42b4-966f-7dcf7504d1f2 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:42:59.946806 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:42:59.967539 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bd161d-4fff-43ee-9c45-059b1379a85d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:59.968251 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bd161d-4fff-43ee-9c45-059b1379a85d None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:59.968529 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bd161d-4fff-43ee-9c45-059b1379a85d None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:59.968804 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bd161d-4fff-43ee-9c45-059b1379a85d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:42:59.995681 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65197a0d-164c-426c-89a6-0ebf5915697f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:42:59.996715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65197a0d-164c-426c-89a6-0ebf5915697f None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:42:59.997098 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65197a0d-164c-426c-89a6-0ebf5915697f None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:42:59.997410 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65197a0d-164c-426c-89a6-0ebf5915697f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:00.049663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bd161d-4fff-43ee-9c45-059b1379a85d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:00.051743 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bd161d-4fff-43ee-9c45-059b1379a85d None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:00.084617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65197a0d-164c-426c-89a6-0ebf5915697f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:00.086106 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65197a0d-164c-426c-89a6-0ebf5915697f None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:00.161228 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 224/442] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:42:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:43:00.161973 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bd161d-4fff-43ee-9c45-059b1379a85d None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:00.164159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:00.185962 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:00.202687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 219/443] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:42:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:43:00.202687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65197a0d-164c-426c-89a6-0ebf5915697f None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:00.282252 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bd161d-4fff-43ee-9c45-059b1379a85d None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:00.321544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65197a0d-164c-426c-89a6-0ebf5915697f None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:00.322849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6d8500-5d05-446b-90df-00e803bd3aff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:00.323489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6d8500-5d05-446b-90df-00e803bd3aff None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:00.323764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6d8500-5d05-446b-90df-00e803bd3aff None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:00.324038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6d8500-5d05-446b-90df-00e803bd3aff None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:00.335917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6d8500-5d05-446b-90df-00e803bd3aff None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:00.340869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-2a6d8500-5d05-446b-90df-00e803bd3aff None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:43:00.340869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:00.340869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:00.340869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:00.340869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:00.340869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:00.340869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:00.340869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:00.340869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:00.340869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:00.340869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:00.340869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:00.340869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:00.340869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:00.340869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:00.340869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:00.342694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:00.342694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:00.342694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:00.342694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:00.342694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:00.342694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:00.342694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:00.342694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:00.342694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:00.342694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:00.342694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:00.342694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:00.342694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:00.342694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:00.342694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:00.342694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:00.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:00.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:00.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:00.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:00.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:00.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:00.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:00.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:00.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:00.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:00.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:43:00.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:00.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 225/444] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:43:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:00.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6d8500-5d05-446b-90df-00e803bd3aff None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:00.363740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9493a91-8f91-4eba-b84b-64409869a0fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:00.364382 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9493a91-8f91-4eba-b84b-64409869a0fc None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:00.364661 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9493a91-8f91-4eba-b84b-64409869a0fc None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:00.364932 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9493a91-8f91-4eba-b84b-64409869a0fc None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:00.366067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9493a91-8f91-4eba-b84b-64409869a0fc None admin] RBAC: Authorizing `identity:get_project(project_id=8085b2496c6847aa83e809bde1f3e986)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:00.367566 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9493a91-8f91-4eba-b84b-64409869a0fc None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:00.369206 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 220/445] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:43:00 2022] GET /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986 => generated 302 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:00.369977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9493a91-8f91-4eba-b84b-64409869a0fc None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:00.414751 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:00.435501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:00.462093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6d8500-5d05-446b-90df-00e803bd3aff None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:00.481765 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9493a91-8f91-4eba-b84b-64409869a0fc None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:00.500271 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d34e28-5268-4b01-bb26-faea780a811b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:00.500983 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d34e28-5268-4b01-bb26-faea780a811b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:00.501334 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d34e28-5268-4b01-bb26-faea780a811b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:00.501666 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d34e28-5268-4b01-bb26-faea780a811b None admin] PATH_INFO: `/v3/projects/8085b2496c6847aa83e809bde1f3e986` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:00.503011 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d34e28-5268-4b01-bb26-faea780a811b None admin] RBAC: Authorizing `identity:get_project(project_id=8085b2496c6847aa83e809bde1f3e986)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:00.504616 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d34e28-5268-4b01-bb26-faea780a811b None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:00.506675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 226/446] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:43:00 2022] GET /identity/v3/projects/8085b2496c6847aa83e809bde1f3e986 => generated 302 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:00.507588 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d34e28-5268-4b01-bb26-faea780a811b None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:00.526100 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a49c19-1fca-4bcb-bba5-f674fdfd9ab0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:00.526754 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a49c19-1fca-4bcb-bba5-f674fdfd9ab0 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:00.527097 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a49c19-1fca-4bcb-bba5-f674fdfd9ab0 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:00.527933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a49c19-1fca-4bcb-bba5-f674fdfd9ab0 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:00.528611 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a49c19-1fca-4bcb-bba5-f674fdfd9ab0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:00.528913 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a49c19-1fca-4bcb-bba5-f674fdfd9ab0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:00.530308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a49c19-1fca-4bcb-bba5-f674fdfd9ab0 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:00.540074 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 221/447] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:43:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:00.541277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a49c19-1fca-4bcb-bba5-f674fdfd9ab0 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:00.623901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d34e28-5268-4b01-bb26-faea780a811b None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:00.649666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:00.654217 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a49c19-1fca-4bcb-bba5-f674fdfd9ab0 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:00.665798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef5df38-a46f-426e-a43a-bf8dccc34839 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:00.666659 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef5df38-a46f-426e-a43a-bf8dccc34839 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:00.667038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef5df38-a46f-426e-a43a-bf8dccc34839 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:00.667399 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef5df38-a46f-426e-a43a-bf8dccc34839 None admin] PATH_INFO: `/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:00.669378 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef5df38-a46f-426e-a43a-bf8dccc34839 None admin] RBAC: Authorizing `identity:get_group(group_id=cde72a6c511d455a8d0c64ae4d97ef26)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:00.671032 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef5df38-a46f-426e-a43a-bf8dccc34839 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:00.672837 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 227/448] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 18 00:43:00 2022] GET /identity/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26 => generated 226 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:00.673887 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef5df38-a46f-426e-a43a-bf8dccc34839 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:00.691285 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:00.694629 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df626f7c-52e8-4724-8b0c-9c82488f4137 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:00.695280 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df626f7c-52e8-4724-8b0c-9c82488f4137 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:00.695572 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df626f7c-52e8-4724-8b0c-9c82488f4137 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:00.695851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df626f7c-52e8-4724-8b0c-9c82488f4137 None admin] PATH_INFO: `/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:00.697005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df626f7c-52e8-4724-8b0c-9c82488f4137 None admin] RBAC: Authorizing `identity:get_group(group_id=cde72a6c511d455a8d0c64ae4d97ef26)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:00.698492 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df626f7c-52e8-4724-8b0c-9c82488f4137 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:00.707313 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 222/449] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 18 00:43:00 2022] GET /identity/v3/groups/cde72a6c511d455a8d0c64ae4d97ef26 => generated 226 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:00.707313 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df626f7c-52e8-4724-8b0c-9c82488f4137 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:00.786905 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef5df38-a46f-426e-a43a-bf8dccc34839 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:00.808892 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df626f7c-52e8-4724-8b0c-9c82488f4137 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:00.823624 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b62d40e-6aed-4b6e-85a7-e7be51a73d4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:00.824713 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b62d40e-6aed-4b6e-85a7-e7be51a73d4f None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:00.824982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b62d40e-6aed-4b6e-85a7-e7be51a73d4f None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:00.825249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b62d40e-6aed-4b6e-85a7-e7be51a73d4f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:00.843962 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd1491b-eb20-4f88-9b30-b7a382f2137e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:00.844726 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd1491b-eb20-4f88-9b30-b7a382f2137e None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:00.845090 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd1491b-eb20-4f88-9b30-b7a382f2137e None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:00.845449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd1491b-eb20-4f88-9b30-b7a382f2137e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:00.846273 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd1491b-eb20-4f88-9b30-b7a382f2137e None admin] RBAC: Adding query filter params (group.id=cde72a6c511d455a8d0c64ae4d97ef26, role.id=18f6cee478e54567b70d5c7557fe694e, scope.project.id=8085b2496c6847aa83e809bde1f3e986) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:00.846717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd1491b-eb20-4f88-9b30-b7a382f2137e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:00.848242 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd1491b-eb20-4f88-9b30-b7a382f2137e None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:00.855165 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 223/450] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Sep 18 00:43:00 2022] GET /identity/v3/role_assignments?group.id=cde72a6c511d455a8d0c64ae4d97ef26&scope.project.id=8085b2496c6847aa83e809bde1f3e986&role.id=18f6cee478e54567b70d5c7557fe694e => generated 611 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:00.856297 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd1491b-eb20-4f88-9b30-b7a382f2137e None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:00.898727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:00.920929 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b62d40e-6aed-4b6e-85a7-e7be51a73d4f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:00.922373 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b62d40e-6aed-4b6e-85a7-e7be51a73d4f None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:00.951947 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:01.005632 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd1491b-eb20-4f88-9b30-b7a382f2137e None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:01.053417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c86450b-b98f-4400-ac24-eeb42fe4be36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:01.054035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c86450b-b98f-4400-ac24-eeb42fe4be36 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:01.054308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c86450b-b98f-4400-ac24-eeb42fe4be36 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:01.054670 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c86450b-b98f-4400-ac24-eeb42fe4be36 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:01.055415 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c86450b-b98f-4400-ac24-eeb42fe4be36 None admin] RBAC: Adding query filter params (group.id=cde72a6c511d455a8d0c64ae4d97ef26, role.id=cf934cf252ce42d5a0445d3e156348d1, scope.project.id=8085b2496c6847aa83e809bde1f3e986) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:01.055727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c86450b-b98f-4400-ac24-eeb42fe4be36 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:01.057920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c86450b-b98f-4400-ac24-eeb42fe4be36 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:01.064769 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 224/451] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Sep 18 00:43:00 2022] GET /identity/v3/role_assignments?group.id=cde72a6c511d455a8d0c64ae4d97ef26&scope.project.id=8085b2496c6847aa83e809bde1f3e986&role.id=cf934cf252ce42d5a0445d3e156348d1 => generated 611 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:01.079731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 228/452] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:43:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:43:01.085845 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c86450b-b98f-4400-ac24-eeb42fe4be36 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:01.229280 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:01.238349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c86450b-b98f-4400-ac24-eeb42fe4be36 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:01.289525 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb20963b-2a27-42d7-948a-16c7c2f89b97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:01.290573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb20963b-2a27-42d7-948a-16c7c2f89b97 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:01.291028 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb20963b-2a27-42d7-948a-16c7c2f89b97 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:01.291440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb20963b-2a27-42d7-948a-16c7c2f89b97 None admin] PATH_INFO: `/v3/projects/2e41e0644d514666af8ca7ed05ebaec8` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:01.293509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb20963b-2a27-42d7-948a-16c7c2f89b97 None admin] RBAC: Authorizing `identity:get_project(project_id=2e41e0644d514666af8ca7ed05ebaec8)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:01.295888 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb20963b-2a27-42d7-948a-16c7c2f89b97 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:01.299174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 225/453] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:43:01 2022] GET /identity/v3/projects/2e41e0644d514666af8ca7ed05ebaec8 => generated 344 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:01.305443 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b62d40e-6aed-4b6e-85a7-e7be51a73d4f None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:01.427906 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b62d40e-6aed-4b6e-85a7-e7be51a73d4f None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:01.447159 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:01.472024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5931a5d2-e806-4738-a43a-6a4e2b35cff1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:01.472667 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5931a5d2-e806-4738-a43a-6a4e2b35cff1 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:01.472944 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5931a5d2-e806-4738-a43a-6a4e2b35cff1 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:01.473208 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5931a5d2-e806-4738-a43a-6a4e2b35cff1 None admin] PATH_INFO: `/v3/groups/6af07276afdc4cebab987faea681089d` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:01.474357 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5931a5d2-e806-4738-a43a-6a4e2b35cff1 None admin] RBAC: Authorizing `identity:get_group(group_id=6af07276afdc4cebab987faea681089d)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:01.475882 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5931a5d2-e806-4738-a43a-6a4e2b35cff1 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:01.477758 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 229/454] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 18 00:43:01 2022] GET /identity/v3/groups/6af07276afdc4cebab987faea681089d => generated 229 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:01.485753 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb20963b-2a27-42d7-948a-16c7c2f89b97 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:01.589460 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb20963b-2a27-42d7-948a-16c7c2f89b97 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:01.622799 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ec3883-ee30-4c6b-941c-7563b0019454 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:01.623400 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ec3883-ee30-4c6b-941c-7563b0019454 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:01.623676 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ec3883-ee30-4c6b-941c-7563b0019454 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:01.623945 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ec3883-ee30-4c6b-941c-7563b0019454 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:01.624672 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ec3883-ee30-4c6b-941c-7563b0019454 None admin] RBAC: Adding query filter params (group.id=6af07276afdc4cebab987faea681089d, role.id=1e04c9fea65244d28e1e59fc4c64f32f, scope.project.id=2e41e0644d514666af8ca7ed05ebaec8) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:01.624976 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ec3883-ee30-4c6b-941c-7563b0019454 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:01.626406 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ec3883-ee30-4c6b-941c-7563b0019454 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:01.632314 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 226/455] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Sep 18 00:43:01 2022] GET /identity/v3/role_assignments?group.id=6af07276afdc4cebab987faea681089d&scope.project.id=2e41e0644d514666af8ca7ed05ebaec8&role.id=1e04c9fea65244d28e1e59fc4c64f32f => generated 611 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:04.492435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8019fdf-9e03-496a-9c86-f75d5c661e21 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:04.492982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8019fdf-9e03-496a-9c86-f75d5c661e21 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:04.492982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8019fdf-9e03-496a-9c86-f75d5c661e21 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:04.494586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 230/456] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:04.500316 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f29c3f-4b87-41ab-9dbe-91c71fe74d3e None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:04.500786 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f29c3f-4b87-41ab-9dbe-91c71fe74d3e None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:04.500871 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f29c3f-4b87-41ab-9dbe-91c71fe74d3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:04.531022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-80f29c3f-4b87-41ab-9dbe-91c71fe74d3e None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:04.532267 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-80f29c3f-4b87-41ab-9dbe-91c71fe74d3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:04.573809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:04.658918 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 227/457] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:43:05.045688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc77857-434d-4ae0-8ee8-e45993077b4e None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:05.046389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc77857-434d-4ae0-8ee8-e45993077b4e None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:05.046898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc77857-434d-4ae0-8ee8-e45993077b4e None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:05.048707 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 231/458] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:05.064516 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975bae77-053a-4ac5-a7ec-cf6054e38160 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:05.065123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975bae77-053a-4ac5-a7ec-cf6054e38160 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:05.065613 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975bae77-053a-4ac5-a7ec-cf6054e38160 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:05.095622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-975bae77-053a-4ac5-a7ec-cf6054e38160 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:05.097187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-975bae77-053a-4ac5-a7ec-cf6054e38160 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:05.193337 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 228/459] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:43:05.220820 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc77857-434d-4ae0-8ee8-e45993077b4e None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:05.359737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc77857-434d-4ae0-8ee8-e45993077b4e None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:05.406895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c437e0c5-13f3-427c-80aa-80b1a9cf632c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:05.407556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c437e0c5-13f3-427c-80aa-80b1a9cf632c None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:05.407895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c437e0c5-13f3-427c-80aa-80b1a9cf632c None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:05.408176 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c437e0c5-13f3-427c-80aa-80b1a9cf632c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:05.425838 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c437e0c5-13f3-427c-80aa-80b1a9cf632c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:05.427360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c437e0c5-13f3-427c-80aa-80b1a9cf632c None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:05.451764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-c437e0c5-13f3-427c-80aa-80b1a9cf632c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:05.451764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:05.451764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:05.451764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:05.451764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:05.451764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:05.451764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:05.451764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:05.451764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:05.451764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:05.451764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:05.451764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:05.451764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:05.451764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:05.451764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:05.451764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:05.453974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:05.453974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:05.453974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:05.453974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:05.453974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:05.453974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:05.453974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:05.453974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:05.453974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:05.453974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:05.453974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:05.453974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:05.453974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:05.453974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:05.453974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:05.453974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:05.453974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:05.456079 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:05.456079 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:05.456079 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:05.456079 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 232/460] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:05.462327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975bae77-053a-4ac5-a7ec-cf6054e38160 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:05.573743 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c437e0c5-13f3-427c-80aa-80b1a9cf632c None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:05.643766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:05.707019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:05.726007 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975bae77-053a-4ac5-a7ec-cf6054e38160 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:05.797063 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c437e0c5-13f3-427c-80aa-80b1a9cf632c None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:05.860042 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eae4faa-63e5-408e-999e-fb01286f6ec9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:05.860688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eae4faa-63e5-408e-999e-fb01286f6ec9 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:05.860973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eae4faa-63e5-408e-999e-fb01286f6ec9 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:05.861248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eae4faa-63e5-408e-999e-fb01286f6ec9 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:05.861935 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eae4faa-63e5-408e-999e-fb01286f6ec9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:05.862244 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eae4faa-63e5-408e-999e-fb01286f6ec9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:05.863727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eae4faa-63e5-408e-999e-fb01286f6ec9 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:05.886321 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019f41a0-3a57-4913-87c7-31295781b694 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:05.887972 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019f41a0-3a57-4913-87c7-31295781b694 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:05.888695 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019f41a0-3a57-4913-87c7-31295781b694 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:05.889730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019f41a0-3a57-4913-87c7-31295781b694 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:05.909298 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 229/461] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 445 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:05.916493 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eae4faa-63e5-408e-999e-fb01286f6ec9 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:05.918152 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-019f41a0-3a57-4913-87c7-31295781b694 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:05.920689 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-019f41a0-3a57-4913-87c7-31295781b694 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:05.960800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-019f41a0-3a57-4913-87c7-31295781b694 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:05.960800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:05.960800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:05.960800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:05.960800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:05.960800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:05.960800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:05.960800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:05.960800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:05.960800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:05.960800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:05.960800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:05.960800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:05.960800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:05.960800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:05.960800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:05.962706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:05.962706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:05.962706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:05.962706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:05.962706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:05.962706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:05.962706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:05.962706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:05.962706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:05.962706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:05.962706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:05.962706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:05.962706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:05.962706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:05.962706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:05.962706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:05.962706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:05.964364 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:05.964364 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:05.964364 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:05.967877 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 233/462] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 394 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:05.973662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019f41a0-3a57-4913-87c7-31295781b694 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:06.091597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:06.103105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019f41a0-3a57-4913-87c7-31295781b694 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:06.141102 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d40acc9-75ef-4923-8dea-bbe94ed7ca7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:06.141878 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d40acc9-75ef-4923-8dea-bbe94ed7ca7b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:06.142268 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d40acc9-75ef-4923-8dea-bbe94ed7ca7b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:06.142674 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d40acc9-75ef-4923-8dea-bbe94ed7ca7b None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:06.143442 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d40acc9-75ef-4923-8dea-bbe94ed7ca7b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:06.143849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d40acc9-75ef-4923-8dea-bbe94ed7ca7b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:06.145033 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eae4faa-63e5-408e-999e-fb01286f6ec9 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:06.145465 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d40acc9-75ef-4923-8dea-bbe94ed7ca7b None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:06.166022 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 234/463] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:06.167589 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e289b811-46d5-4abb-8a90-6ffefedd7da8 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:06.168029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e289b811-46d5-4abb-8a90-6ffefedd7da8 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:06.168406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e289b811-46d5-4abb-8a90-6ffefedd7da8 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:06.170256 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 235/464] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:06.171110 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e289b811-46d5-4abb-8a90-6ffefedd7da8 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:06.226716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1240bf2-2188-4205-8437-a038c72591fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:06.228323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1240bf2-2188-4205-8437-a038c72591fd None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:06.229479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1240bf2-2188-4205-8437-a038c72591fd None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:06.230143 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1240bf2-2188-4205-8437-a038c72591fd None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:06.232362 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1240bf2-2188-4205-8437-a038c72591fd None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:06.235849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1240bf2-2188-4205-8437-a038c72591fd None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:06.285018 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 230/465] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 18 00:43:05 2022] POST /identity/v3/users => generated 248 bytes in 370 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:43:06.288717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df034cfb-e6d1-459d-ac27-a619770d5dd4 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:06.289934 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df034cfb-e6d1-459d-ac27-a619770d5dd4 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:06.291032 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df034cfb-e6d1-459d-ac27-a619770d5dd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:06.309911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e289b811-46d5-4abb-8a90-6ffefedd7da8 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:06.311260 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:06.338107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-df034cfb-e6d1-459d-ac27-a619770d5dd4 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:06.340911 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-df034cfb-e6d1-459d-ac27-a619770d5dd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:06.403157 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84cdaf7-740f-4fc2-b406-1dcc8a66e5ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:06.404114 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84cdaf7-740f-4fc2-b406-1dcc8a66e5ec None admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:06.404578 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84cdaf7-740f-4fc2-b406-1dcc8a66e5ec None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:06.405033 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84cdaf7-740f-4fc2-b406-1dcc8a66e5ec None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:06.406282 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84cdaf7-740f-4fc2-b406-1dcc8a66e5ec None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:06.408842 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84cdaf7-740f-4fc2-b406-1dcc8a66e5ec None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:06.479721 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 236/466] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 18 00:43:06 2022] POST /identity/v3/users => generated 246 bytes in 308 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:43:06.534837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 231/467] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 248 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:43:06.911718 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84cdaf7-740f-4fc2-b406-1dcc8a66e5ec None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:06.917508 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:06.979274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbc0c29-e255-4071-812b-d33d68ff0fd8 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:06.979822 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbc0c29-e255-4071-812b-d33d68ff0fd8 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:06.980866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbc0c29-e255-4071-812b-d33d68ff0fd8 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:06.983734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 232/468] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:06 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:06.991424 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fafbbff-f823-4de4-bb94-a3bcf82783c5 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:06.991424 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fafbbff-f823-4de4-bb94-a3bcf82783c5 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:06.991424 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fafbbff-f823-4de4-bb94-a3bcf82783c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:07.016502 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-5fafbbff-f823-4de4-bb94-a3bcf82783c5 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:07.019823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-5fafbbff-f823-4de4-bb94-a3bcf82783c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:07.085213 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84cdaf7-740f-4fc2-b406-1dcc8a66e5ec None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:07.134686 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c8cace-c261-4b20-a208-a7b82c018336 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:07.136127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c8cace-c261-4b20-a208-a7b82c018336 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:07.136878 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c8cace-c261-4b20-a208-a7b82c018336 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:07.137609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c8cace-c261-4b20-a208-a7b82c018336 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:07.158978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c8cace-c261-4b20-a208-a7b82c018336 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:07.162268 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c8cace-c261-4b20-a208-a7b82c018336 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:07.193754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-59c8cace-c261-4b20-a208-a7b82c018336 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:07.193754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:07.193754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:07.193754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:07.193754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:07.193754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:07.193754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:07.193754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:07.193754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:07.193754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:07.193754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:07.193754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:07.193754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:07.193754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:07.193754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:07.193754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:07.195531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:07.195531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:07.195531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:07.195531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:07.195531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:07.195531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:07.195531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:07.195531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:07.195531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:07.195531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:07.195531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:07.195531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:07.195531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:07.195531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:07.195531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:07.195531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:07.195531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:07.197217 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:07.197217 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:07.197217 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:07.200880 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 237/469] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:07.210879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c8cace-c261-4b20-a208-a7b82c018336 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:07.221927 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 233/470] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:43:07.367778 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:07.479280 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c8cace-c261-4b20-a208-a7b82c018336 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:07.552652 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a05222c-0042-4dff-9f09-45f8ec202bff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:07.553287 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a05222c-0042-4dff-9f09-45f8ec202bff None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:07.553567 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a05222c-0042-4dff-9f09-45f8ec202bff None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:07.553838 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a05222c-0042-4dff-9f09-45f8ec202bff None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:07.554533 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a05222c-0042-4dff-9f09-45f8ec202bff None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:07.554839 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a05222c-0042-4dff-9f09-45f8ec202bff None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:07.556232 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a05222c-0042-4dff-9f09-45f8ec202bff None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:07.580431 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 238/471] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 371 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:07.586970 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887e8bdb-8a1b-42c9-82cd-41da0ded2d0b None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:07.590688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a05222c-0042-4dff-9f09-45f8ec202bff None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:07.591635 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887e8bdb-8a1b-42c9-82cd-41da0ded2d0b None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:07.592422 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887e8bdb-8a1b-42c9-82cd-41da0ded2d0b None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:07.599255 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 234/472] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:07 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:07.601849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e11114c-b9ce-4162-b46c-64d61fe20b0d None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:07.602667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e11114c-b9ce-4162-b46c-64d61fe20b0d None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:07.603436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e11114c-b9ce-4162-b46c-64d61fe20b0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:07.657537 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-1e11114c-b9ce-4162-b46c-64d61fe20b0d None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:07.662594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-1e11114c-b9ce-4162-b46c-64d61fe20b0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:07.763523 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:07.775471 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 235/473] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:43:07.776500 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e11114c-b9ce-4162-b46c-64d61fe20b0d None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:07.786331 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a05222c-0042-4dff-9f09-45f8ec202bff None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:07.842448 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:07.915352 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1870d63a-9c0c-4b3e-9658-169430c8f210 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:07.916340 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1870d63a-9c0c-4b3e-9658-169430c8f210 None admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:07.916340 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1870d63a-9c0c-4b3e-9658-169430c8f210 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:07.916576 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1870d63a-9c0c-4b3e-9658-169430c8f210 None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:07.917189 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1870d63a-9c0c-4b3e-9658-169430c8f210 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:07.919335 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1870d63a-9c0c-4b3e-9658-169430c8f210 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:07.963166 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 239/474] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 18 00:43:07 2022] POST /identity/v3/users => generated 248 bytes in 375 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:43:07.982394 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e11114c-b9ce-4162-b46c-64d61fe20b0d None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:08.060185 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c5792a-983f-4deb-8864-5beeac291ed0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:08.061274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c5792a-983f-4deb-8864-5beeac291ed0 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:08.061763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c5792a-983f-4deb-8864-5beeac291ed0 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:08.062241 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c5792a-983f-4deb-8864-5beeac291ed0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:08.087464 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c5792a-983f-4deb-8864-5beeac291ed0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:08.090890 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c5792a-983f-4deb-8864-5beeac291ed0 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:08.129524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-d4c5792a-983f-4deb-8864-5beeac291ed0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:08.129524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:08.129524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:08.129524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:08.129524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:08.129524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:08.129524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:08.129524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:08.129524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:08.129524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:08.129524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:08.129524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:08.129524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:08.129524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:08.129524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:08.129524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:08.131293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:08.131293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:08.131293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:08.131293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:08.131293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:08.131293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:08.131293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:08.131293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:08.131293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:08.131293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:08.131293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:08.131293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:08.131293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:08.131293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:08.131293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:08.131293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:08.131293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:08.132814 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:08.132814 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:08.132814 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:08.135141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 236/475] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 356 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:08.142571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1870d63a-9c0c-4b3e-9658-169430c8f210 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:08.337734 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1870d63a-9c0c-4b3e-9658-169430c8f210 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:08.401333 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ab120e-d967-4fb4-b5f5-462033e12646 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:08.402928 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ab120e-d967-4fb4-b5f5-462033e12646 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:08.403674 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ab120e-d967-4fb4-b5f5-462033e12646 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:08.404446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ab120e-d967-4fb4-b5f5-462033e12646 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:08.405868 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c5792a-983f-4deb-8864-5beeac291ed0 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:08.406341 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38ab120e-d967-4fb4-b5f5-462033e12646 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:08.407192 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38ab120e-d967-4fb4-b5f5-462033e12646 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:08.409914 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38ab120e-d967-4fb4-b5f5-462033e12646 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:08.434310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 240/476] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:08.442827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ab120e-d967-4fb4-b5f5-462033e12646 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:08.481348 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:08.485105 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:08.576253 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c5792a-983f-4deb-8864-5beeac291ed0 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:08.614204 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ab120e-d967-4fb4-b5f5-462033e12646 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:08.619061 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a337c563-8ae3-4400-992c-350b7ec20aaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:08.619685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a337c563-8ae3-4400-992c-350b7ec20aaf None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:08.619966 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a337c563-8ae3-4400-992c-350b7ec20aaf None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:08.620241 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a337c563-8ae3-4400-992c-350b7ec20aaf None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:08.621462 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a337c563-8ae3-4400-992c-350b7ec20aaf None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:08.622936 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a337c563-8ae3-4400-992c-350b7ec20aaf None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:08.632871 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.sql.core [None req-a337c563-8ae3-4400-992c-350b7ec20aaf None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 18 00:43:08.632871 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 18 00:43:08.632871 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [parameters: {'id': '62100c9e6cd5486da277a3703b3a60d1', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 18 00:43:08.632871 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81529) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 18 00:43:08.651506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-a337c563-8ae3-4400-992c-350b7ec20aaf None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 18 00:43:08.651506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:08.651506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 18 00:43:08.651506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 18 00:43:08.651506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 18 00:43:08.651506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 18 00:43:08.651506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 18 00:43:08.651506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application result = self._query(query) Sep 18 00:43:08.651506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 18 00:43:08.651506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application conn.query(q) Sep 18 00:43:08.651506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 18 00:43:08.651506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 18 00:43:08.651506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 18 00:43:08.651506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application result.read() Sep 18 00:43:08.651506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 18 00:43:08.652981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 18 00:43:08.652981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 18 00:43:08.652981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 18 00:43:08.652981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 18 00:43:08.652981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 18 00:43:08.652981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 18 00:43:08.652981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 18 00:43:08.652981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 18 00:43:08.652981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:08.652981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 18 00:43:08.652981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:08.652981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:08.652981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 18 00:43:08.652981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 18 00:43:08.652981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Sep 18 00:43:08.652981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return ref.to_dict() Sep 18 00:43:08.652981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 18 00:43:08.652981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application next(self.gen) Sep 18 00:43:08.653895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 18 00:43:08.653895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application yield resource Sep 18 00:43:08.653895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 18 00:43:08.653895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application next(self.gen) Sep 18 00:43:08.653895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 18 00:43:08.653895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self.session.rollback() Sep 18 00:43:08.653895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 18 00:43:08.653895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self.force_reraise() Sep 18 00:43:08.653895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 18 00:43:08.653895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise self.value Sep 18 00:43:08.653895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 18 00:43:08.653895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 18 00:43:08.653895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 18 00:43:08.653895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application session.commit() Sep 18 00:43:08.653895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 18 00:43:08.653895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 18 00:43:08.653895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 18 00:43:08.654819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._prepare_impl() Sep 18 00:43:08.654819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 18 00:43:08.654819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self.session.flush() Sep 18 00:43:08.654819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 18 00:43:08.654819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._flush(objects) Sep 18 00:43:08.654819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 18 00:43:08.654819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 18 00:43:08.654819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 18 00:43:08.654819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application compat.raise_( Sep 18 00:43:08.654819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 18 00:43:08.654819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception Sep 18 00:43:08.654819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 18 00:43:08.654819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application flush_context.execute() Sep 18 00:43:08.654819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 18 00:43:08.654819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rec.execute(self) Sep 18 00:43:08.654819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 18 00:43:08.654819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 18 00:43:08.655735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 18 00:43:08.655735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 18 00:43:08.655735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 18 00:43:08.655735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 18 00:43:08.655735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 18 00:43:08.655735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 18 00:43:08.655735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 18 00:43:08.655735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 18 00:43:08.655735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 18 00:43:08.655735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 18 00:43:08.655735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 18 00:43:08.655735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 18 00:43:08.655735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 18 00:43:08.655735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 18 00:43:08.655735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 18 00:43:08.655735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception Sep 18 00:43:08.657725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 18 00:43:08.657725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 18 00:43:08.657725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 18 00:43:08.657725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 18 00:43:08.657725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 18 00:43:08.657725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application result = self._query(query) Sep 18 00:43:08.657725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 18 00:43:08.657725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application conn.query(q) Sep 18 00:43:08.657725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 18 00:43:08.657725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 18 00:43:08.657725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 18 00:43:08.657725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application result.read() Sep 18 00:43:08.657725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 18 00:43:08.657725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 18 00:43:08.657725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 18 00:43:08.657725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 18 00:43:08.659056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 18 00:43:08.659056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 18 00:43:08.659056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 18 00:43:08.659056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 18 00:43:08.659056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 18 00:43:08.659056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 18 00:43:08.659056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [parameters: {'id': '62100c9e6cd5486da277a3703b3a60d1', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 18 00:43:08.659056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 18 00:43:08.659056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:08.659056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:43:08.659056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:08.659056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:08.659056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:08.659056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:08.659056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:08.659056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:08.660027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:08.660027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:08.660027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:08.660027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:08.660027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:08.660027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:08.660027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Sep 18 00:43:08.660027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Sep 18 00:43:08.660027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:08.660027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:08.660027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Sep 18 00:43:08.660027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Sep 18 00:43:08.660027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 18 00:43:08.660027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 18 00:43:08.660027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 18 00:43:08.660027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:08.661015 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 237/477] 10.222.0.42 () {60 vars in 1131 bytes} [Sun Sep 18 00:43:08 2022] POST /identity/v3/roles => generated 144 bytes in 248 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:08.662027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a337c563-8ae3-4400-992c-350b7ec20aaf None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:08.675516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ba97d3-3abc-43d1-b28c-f044480a71a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:08.676539 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ba97d3-3abc-43d1-b28c-f044480a71a4 None admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:08.676970 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ba97d3-3abc-43d1-b28c-f044480a71a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:08.677394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ba97d3-3abc-43d1-b28c-f044480a71a4 None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:08.678324 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ba97d3-3abc-43d1-b28c-f044480a71a4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:08.679974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ba97d3-3abc-43d1-b28c-f044480a71a4 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:08.723689 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 241/478] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 18 00:43:08 2022] POST /identity/v3/users => generated 249 bytes in 281 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:43:08.804911 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a337c563-8ae3-4400-992c-350b7ec20aaf None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:08.821774 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:08.863090 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f43365c-2910-47c7-8bb8-ef6c33568c50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:08.863984 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f43365c-2910-47c7-8bb8-ef6c33568c50 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:08.864483 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f43365c-2910-47c7-8bb8-ef6c33568c50 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:08.864898 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f43365c-2910-47c7-8bb8-ef6c33568c50 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:08.883617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f43365c-2910-47c7-8bb8-ef6c33568c50 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:08.889551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-0f43365c-2910-47c7-8bb8-ef6c33568c50 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:43:08.889551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:08.889551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:08.889551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:08.889551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:08.889551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:08.889551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:08.889551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:08.889551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:08.889551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:08.889551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:08.889551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:08.889551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:08.889551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:08.889551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:08.889551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:08.894867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:08.894867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:08.894867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:08.894867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:08.894867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:08.894867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:08.894867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:08.894867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:08.894867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:08.894867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:08.894867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:08.894867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:08.894867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:08.894867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:08.894867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:08.894867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:08.895831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:08.895831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:08.895831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:08.895831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:08.895831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:08.895831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:08.895831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:08.895831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:08.895831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:08.895831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:08.895831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 18 00:43:08.895831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:08.898346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 238/479] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 18 00:43:08 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:08.904817 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ba97d3-3abc-43d1-b28c-f044480a71a4 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:09.013758 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:09.042136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ba97d3-3abc-43d1-b28c-f044480a71a4 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:09.097649 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61562fe-036e-4376-91cf-7b29abfbb4fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:09.098295 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61562fe-036e-4376-91cf-7b29abfbb4fe None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:09.098609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61562fe-036e-4376-91cf-7b29abfbb4fe None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:09.098892 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61562fe-036e-4376-91cf-7b29abfbb4fe None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:09.099800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61562fe-036e-4376-91cf-7b29abfbb4fe None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:09.100201 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61562fe-036e-4376-91cf-7b29abfbb4fe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:09.101578 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61562fe-036e-4376-91cf-7b29abfbb4fe None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:09.113818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 242/480] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:08 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:09.933010 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c319b4a-723f-4eac-9d1f-58fec3ef7b91 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:09.933606 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c319b4a-723f-4eac-9d1f-58fec3ef7b91 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:09.934038 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c319b4a-723f-4eac-9d1f-58fec3ef7b91 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:09.936975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 239/481] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:09 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:09.945882 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350680ed-48fd-4f30-b846-e2ac6749fdd3 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:09.946366 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350680ed-48fd-4f30-b846-e2ac6749fdd3 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:09.946903 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350680ed-48fd-4f30-b846-e2ac6749fdd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:09.977159 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-350680ed-48fd-4f30-b846-e2ac6749fdd3 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:09.979182 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-350680ed-48fd-4f30-b846-e2ac6749fdd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:10.126259 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 243/482] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:43:10.132775 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cd1ace-2455-449b-9c32-0a6f7121cfa1 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:10.133247 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cd1ace-2455-449b-9c32-0a6f7121cfa1 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:10.133728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cd1ace-2455-449b-9c32-0a6f7121cfa1 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:10.136663 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 240/483] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:10 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:10.144951 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9df7fc-6e78-48f8-a676-4542ea30bcaf None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:10.145394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9df7fc-6e78-48f8-a676-4542ea30bcaf None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:10.145843 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9df7fc-6e78-48f8-a676-4542ea30bcaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:10.177967 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-2b9df7fc-6e78-48f8-a676-4542ea30bcaf None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:10.179944 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-2b9df7fc-6e78-48f8-a676-4542ea30bcaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:10.320692 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 244/484] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:43:10.837437 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cd1ace-2455-449b-9c32-0a6f7121cfa1 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:10.844517 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9df7fc-6e78-48f8-a676-4542ea30bcaf None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:10.854088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:11.012036 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cd1ace-2455-449b-9c32-0a6f7121cfa1 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:11.057864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9df7fc-6e78-48f8-a676-4542ea30bcaf None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:11.088336 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b561c2-aec9-4bfb-94aa-8944a0aa22c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:11.088888 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b561c2-aec9-4bfb-94aa-8944a0aa22c1 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:11.089170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b561c2-aec9-4bfb-94aa-8944a0aa22c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:11.089437 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b561c2-aec9-4bfb-94aa-8944a0aa22c1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:11.123502 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b561c2-aec9-4bfb-94aa-8944a0aa22c1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:11.128766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-50b561c2-aec9-4bfb-94aa-8944a0aa22c1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:11.128766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:11.128766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:11.128766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:11.128766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:11.128766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:11.128766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:11.128766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:11.128766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:11.128766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:11.128766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:11.128766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:11.128766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:11.128766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:11.128766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:11.128766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:11.129809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:11.129809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:11.129809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:11.129809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:11.129809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:11.129809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:11.129809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:11.129809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:11.129809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:11.129809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:11.129809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:11.129809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:11.129809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:11.129809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:11.129809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:11.129809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:11.130719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:11.130719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:11.130719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:11.130719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:11.130719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:11.130719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:11.130719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:11.130719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:11.130719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:11.130719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:11.130719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:11.130719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:11.131387 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b98a480-43fd-434f-a7fb-6d7c84271f6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:11.132858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b98a480-43fd-434f-a7fb-6d7c84271f6a None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:11.133640 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b98a480-43fd-434f-a7fb-6d7c84271f6a None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:11.134372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b98a480-43fd-434f-a7fb-6d7c84271f6a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:11.140061 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 241/485] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:43:10 2022] GET /identity/v3/roles/service => generated 85 bytes in 304 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:11.148762 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b561c2-aec9-4bfb-94aa-8944a0aa22c1 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:11.156780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b98a480-43fd-434f-a7fb-6d7c84271f6a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:11.167276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-7b98a480-43fd-434f-a7fb-6d7c84271f6a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:11.167276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:11.167276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:11.167276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:11.167276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:11.167276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:11.167276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:11.167276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:11.167276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:11.167276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:11.167276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:11.167276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:11.167276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:11.167276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:11.167276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:11.167276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:11.169082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:11.169082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:11.169082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:11.169082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:11.169082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:11.169082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:11.169082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:11.169082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:11.169082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:11.169082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:11.169082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:11.169082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:11.169082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:11.169082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:11.169082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:11.169082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:11.170691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:11.170691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:11.170691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:11.170691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:11.170691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:11.170691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:11.170691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:11.170691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:11.170691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:11.170691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:11.170691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:11.170691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:11.175697 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 245/486] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:43:10 2022] GET /identity/v3/roles/service => generated 85 bytes in 334 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:11.182553 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b98a480-43fd-434f-a7fb-6d7c84271f6a None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:11.232762 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:11.322392 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:11.388058 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b98a480-43fd-434f-a7fb-6d7c84271f6a None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:11.469135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4647da05-d547-4a61-97dd-3949cd320ee8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:11.470140 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4647da05-d547-4a61-97dd-3949cd320ee8 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:11.470662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4647da05-d547-4a61-97dd-3949cd320ee8 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:11.471145 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4647da05-d547-4a61-97dd-3949cd320ee8 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:11.472509 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4647da05-d547-4a61-97dd-3949cd320ee8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:11.473041 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4647da05-d547-4a61-97dd-3949cd320ee8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:11.475577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4647da05-d547-4a61-97dd-3949cd320ee8 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:11.492272 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 246/487] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 18 00:43:11 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:11.499755 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4647da05-d547-4a61-97dd-3949cd320ee8 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:11.502824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b561c2-aec9-4bfb-94aa-8944a0aa22c1 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:11.575690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe36535b-6840-4f6a-8a6a-df1acd988078 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:11.576318 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe36535b-6840-4f6a-8a6a-df1acd988078 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:11.576597 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe36535b-6840-4f6a-8a6a-df1acd988078 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:11.576874 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe36535b-6840-4f6a-8a6a-df1acd988078 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:11.577565 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe36535b-6840-4f6a-8a6a-df1acd988078 None admin] RBAC: Adding query filter params (name=service) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:11.577870 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe36535b-6840-4f6a-8a6a-df1acd988078 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:11.579331 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe36535b-6840-4f6a-8a6a-df1acd988078 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:11.596232 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 242/488] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 18 00:43:11 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 448 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:43:11.605416 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c389ce38-7133-4d25-a84f-87411177e039 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:11.605632 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c389ce38-7133-4d25-a84f-87411177e039 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:11.605918 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c389ce38-7133-4d25-a84f-87411177e039 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:11.607614 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 243/489] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:11.608661 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c389ce38-7133-4d25-a84f-87411177e039 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:11.715354 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4647da05-d547-4a61-97dd-3949cd320ee8 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:11.731849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:11.744107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c389ce38-7133-4d25-a84f-87411177e039 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:11.775987 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:11.784133 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7491821a-fc7d-49a8-acf5-113c65bea426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:11.784795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7491821a-fc7d-49a8-acf5-113c65bea426 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:11.785078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7491821a-fc7d-49a8-acf5-113c65bea426 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:11.785354 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7491821a-fc7d-49a8-acf5-113c65bea426 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:11.790335 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1969685d-56e5-4aeb-b596-ef0ab5aee383 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:11.791759 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1969685d-56e5-4aeb-b596-ef0ab5aee383 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:11.792795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1969685d-56e5-4aeb-b596-ef0ab5aee383 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:11.794204 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1969685d-56e5-4aeb-b596-ef0ab5aee383 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:11.798193 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7491821a-fc7d-49a8-acf5-113c65bea426 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:11.799788 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7491821a-fc7d-49a8-acf5-113c65bea426 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:11.817174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-7491821a-fc7d-49a8-acf5-113c65bea426 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:11.817174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:11.817174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:11.817174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:11.817174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:11.817174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:11.817174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:11.817174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:11.817174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:11.817174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:11.817174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:11.817174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:11.817174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:11.817174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:11.817174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:11.817174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:11.825377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:11.825377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:11.825377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:11.825377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:11.825377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:11.825377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:11.825377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:11.825377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:11.825377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:11.825377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:11.825377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:11.825377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:11.825377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:11.825377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:11.825377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:11.825377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:11.825377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:11.827035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:11.827035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:11.827035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:11.827035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 244/490] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:11.827035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0d1a91-a2a2-4a79-be39-2a9e3fe164bf None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:11.827035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0d1a91-a2a2-4a79-be39-2a9e3fe164bf None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:11.827035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0d1a91-a2a2-4a79-be39-2a9e3fe164bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:11.827745 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1969685d-56e5-4aeb-b596-ef0ab5aee383 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:11.828653 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1969685d-56e5-4aeb-b596-ef0ab5aee383 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:11.847279 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-4e0d1a91-a2a2-4a79-be39-2a9e3fe164bf None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:11.848560 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-4e0d1a91-a2a2-4a79-be39-2a9e3fe164bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:11.857257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-1969685d-56e5-4aeb-b596-ef0ab5aee383 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:11.857257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:11.857257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:11.857257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:11.857257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:11.857257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:11.857257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:11.857257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:11.857257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:11.857257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:11.857257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:11.857257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:11.857257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:11.857257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:11.857257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:11.857257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:11.860849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:11.860849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:11.860849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:11.860849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:11.860849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:11.860849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:11.860849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:11.860849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:11.860849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:11.860849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:11.860849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:11.860849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:11.860849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:11.860849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:11.860849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:11.860849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:11.860849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:11.867049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:11.867049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:11.867049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:11.867049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 247/491] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 360 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:11.867049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1969685d-56e5-4aeb-b596-ef0ab5aee383 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:11.933749 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 245/492] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:43:11.934747 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0d1a91-a2a2-4a79-be39-2a9e3fe164bf None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:12.000249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1969685d-56e5-4aeb-b596-ef0ab5aee383 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:12.048611 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7e86fd-5a2d-4f51-80a3-15f6b29d3bf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:12.049687 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7e86fd-5a2d-4f51-80a3-15f6b29d3bf9 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:12.050496 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7e86fd-5a2d-4f51-80a3-15f6b29d3bf9 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:12.050970 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7e86fd-5a2d-4f51-80a3-15f6b29d3bf9 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:12.051862 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7e86fd-5a2d-4f51-80a3-15f6b29d3bf9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:12.052480 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7e86fd-5a2d-4f51-80a3-15f6b29d3bf9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:12.054213 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7e86fd-5a2d-4f51-80a3-15f6b29d3bf9 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:12.062876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:12.078839 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 248/493] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:12.080637 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45dcdff-f7cd-4eb2-9c3a-72f180263ee5 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:12.081140 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45dcdff-f7cd-4eb2-9c3a-72f180263ee5 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:12.081660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45dcdff-f7cd-4eb2-9c3a-72f180263ee5 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:12.083584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 249/494] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:12.087359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45dcdff-f7cd-4eb2-9c3a-72f180263ee5 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:12.087976 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0d1a91-a2a2-4a79-be39-2a9e3fe164bf None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:12.125966 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58983e9-c3f6-41bb-a294-192fc559abd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:12.126645 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58983e9-c3f6-41bb-a294-192fc559abd3 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:12.126928 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58983e9-c3f6-41bb-a294-192fc559abd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:12.127272 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58983e9-c3f6-41bb-a294-192fc559abd3 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:12.127919 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58983e9-c3f6-41bb-a294-192fc559abd3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:12.128229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58983e9-c3f6-41bb-a294-192fc559abd3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:12.129655 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58983e9-c3f6-41bb-a294-192fc559abd3 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:12.143170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 246/495] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:12.148519 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d868ffc-da8b-48f5-a1b3-63f656f3e3b6 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:12.148519 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d868ffc-da8b-48f5-a1b3-63f656f3e3b6 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:12.148519 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d868ffc-da8b-48f5-a1b3-63f656f3e3b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:12.149596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:12.165858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-4d868ffc-da8b-48f5-a1b3-63f656f3e3b6 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:12.167125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-4d868ffc-da8b-48f5-a1b3-63f656f3e3b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:12.221362 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45dcdff-f7cd-4eb2-9c3a-72f180263ee5 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:12.242163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 247/496] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:43:12.243110 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d868ffc-da8b-48f5-a1b3-63f656f3e3b6 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:12.265031 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322cea6a-134f-4a2e-af43-953c6405aa0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:12.266071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322cea6a-134f-4a2e-af43-953c6405aa0c None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:12.266681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322cea6a-134f-4a2e-af43-953c6405aa0c None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:12.267191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322cea6a-134f-4a2e-af43-953c6405aa0c None admin] PATH_INFO: `/v3/users/glance` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:12.276823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322cea6a-134f-4a2e-af43-953c6405aa0c None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:12.278761 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322cea6a-134f-4a2e-af43-953c6405aa0c None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:12.295795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-322cea6a-134f-4a2e-af43-953c6405aa0c None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 18 00:43:12.295795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:12.295795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:12.295795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:12.295795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:12.295795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:12.295795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:12.295795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:12.295795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:12.295795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:12.295795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:12.295795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:12.295795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:43:12.295795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:43:12.295795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:43:12.295795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:43:12.296792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:12.296792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:12.296792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:43:12.296792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:12.296792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:43:12.296792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:12.296792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:12.296792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:12.296792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:12.296792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:12.296792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:12.296792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:12.296792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:12.296792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:12.296792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:12.296792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:12.296792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:12.296792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:12.297693 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:12.297693 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:12.297693 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:43:12.297693 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:43:12.297693 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:43:12.297693 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:43:12.297693 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:43:12.297693 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:43:12.297693 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 18 00:43:12.297693 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:12.300642 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 250/497] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:43:12 2022] GET /identity/v3/users/glance => generated 84 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:12.311588 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:12.313034 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322cea6a-134f-4a2e-af43-953c6405aa0c None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:12.359014 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d868ffc-da8b-48f5-a1b3-63f656f3e3b6 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:12.396379 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55477a1d-ac5a-45e2-bdd8-2ffddf1a1d5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:12.396788 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55477a1d-ac5a-45e2-bdd8-2ffddf1a1d5a None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:12.397016 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55477a1d-ac5a-45e2-bdd8-2ffddf1a1d5a None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:12.397293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55477a1d-ac5a-45e2-bdd8-2ffddf1a1d5a None admin] PATH_INFO: `/v3/users/nova` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:12.406041 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55477a1d-ac5a-45e2-bdd8-2ffddf1a1d5a None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:12.407612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55477a1d-ac5a-45e2-bdd8-2ffddf1a1d5a None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:12.422989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-55477a1d-ac5a-45e2-bdd8-2ffddf1a1d5a None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 18 00:43:12.422989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:12.422989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:12.422989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:12.422989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:12.422989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:12.422989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:12.422989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:12.422989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:12.422989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:12.422989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:12.422989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:12.422989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:43:12.422989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:43:12.422989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:43:12.422989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:43:12.423957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:12.423957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:12.423957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:43:12.423957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:12.423957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:43:12.423957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:12.423957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:12.423957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:12.423957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:12.423957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:12.423957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:12.423957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:12.423957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:12.423957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:12.423957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:12.423957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:12.423957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:12.423957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:12.424850 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:12.424850 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:12.424850 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:43:12.424850 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:43:12.424850 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:43:12.424850 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:43:12.424850 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:43:12.424850 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:43:12.424850 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 18 00:43:12.424850 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:12.424850 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 248/498] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Sep 18 00:43:12 2022] GET /identity/v3/users/nova => generated 82 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:12.425681 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55477a1d-ac5a-45e2-bdd8-2ffddf1a1d5a None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:12.445391 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322cea6a-134f-4a2e-af43-953c6405aa0c None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:12.447667 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:12.488488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f10d756-7c0e-428b-a106-785bc8aa79ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:12.489540 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f10d756-7c0e-428b-a106-785bc8aa79ea None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:12.490020 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f10d756-7c0e-428b-a106-785bc8aa79ea None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:12.490542 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f10d756-7c0e-428b-a106-785bc8aa79ea None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:12.491612 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f10d756-7c0e-428b-a106-785bc8aa79ea None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:12.492363 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f10d756-7c0e-428b-a106-785bc8aa79ea None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:12.494998 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f10d756-7c0e-428b-a106-785bc8aa79ea None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:12.539332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 251/499] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Sep 18 00:43:12 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:12.540668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f10d756-7c0e-428b-a106-785bc8aa79ea None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:12.541402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55477a1d-ac5a-45e2-bdd8-2ffddf1a1d5a None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:12.571137 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:12.583477 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48472ee-46e2-4786-b754-b6e318fd25c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:12.584140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48472ee-46e2-4786-b754-b6e318fd25c0 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:12.584429 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48472ee-46e2-4786-b754-b6e318fd25c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:12.584706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48472ee-46e2-4786-b754-b6e318fd25c0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:12.597287 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48472ee-46e2-4786-b754-b6e318fd25c0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:12.602973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-a48472ee-46e2-4786-b754-b6e318fd25c0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:12.602973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:12.602973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:12.602973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:12.602973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:12.602973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:12.602973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:12.602973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:12.602973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:12.602973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:12.602973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:12.602973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:12.602973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:12.602973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:12.602973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:12.602973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:12.603939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:12.603939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:12.603939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:12.603939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:12.603939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:12.603939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:12.603939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:12.603939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:12.603939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:12.603939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:12.603939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:12.603939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:12.603939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:12.603939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:12.603939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:12.603939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:12.609791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:12.609791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:12.609791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:12.609791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:12.609791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:12.609791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:12.609791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:12.609791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:12.609791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:12.609791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:12.609791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:12.609791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:12.609791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 249/500] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:43:12 2022] GET /identity/v3/roles/service => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:12.609791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48472ee-46e2-4786-b754-b6e318fd25c0 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:12.668859 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f10d756-7c0e-428b-a106-785bc8aa79ea None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:12.708423 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24f5a31-c6a7-4236-abce-e95e3d0a0776 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:12.709037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24f5a31-c6a7-4236-abce-e95e3d0a0776 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:12.709315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24f5a31-c6a7-4236-abce-e95e3d0a0776 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:12.709585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24f5a31-c6a7-4236-abce-e95e3d0a0776 None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:12.710331 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24f5a31-c6a7-4236-abce-e95e3d0a0776 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:12.710667 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24f5a31-c6a7-4236-abce-e95e3d0a0776 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:12.712126 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24f5a31-c6a7-4236-abce-e95e3d0a0776 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:12.730802 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48472ee-46e2-4786-b754-b6e318fd25c0 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:12.737459 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 252/501] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Sep 18 00:43:12 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:12.737459 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24f5a31-c6a7-4236-abce-e95e3d0a0776 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:12.758425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:12.780702 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dd9a8a-7450-449b-8ff3-a2a5af4200ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:12.781352 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dd9a8a-7450-449b-8ff3-a2a5af4200ec None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:12.781634 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dd9a8a-7450-449b-8ff3-a2a5af4200ec None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:12.781910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dd9a8a-7450-449b-8ff3-a2a5af4200ec None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:12.850741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:12.867679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24f5a31-c6a7-4236-abce-e95e3d0a0776 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:12.873144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83dd9a8a-7450-449b-8ff3-a2a5af4200ec None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:12.874719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83dd9a8a-7450-449b-8ff3-a2a5af4200ec None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:12.906910 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4b501b-efd4-43b9-94c1-00c26d2a5cc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:12.907567 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4b501b-efd4-43b9-94c1-00c26d2a5cc3 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:12.907979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4b501b-efd4-43b9-94c1-00c26d2a5cc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:12.908133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4b501b-efd4-43b9-94c1-00c26d2a5cc3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:12.923129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4b501b-efd4-43b9-94c1-00c26d2a5cc3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:12.928818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-ce4b501b-efd4-43b9-94c1-00c26d2a5cc3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:12.928818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:12.928818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:12.928818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:12.928818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:12.928818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:12.928818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:12.928818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:12.928818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:12.928818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:12.928818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:12.928818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:12.928818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:12.928818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:12.928818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:12.928818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:12.929794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:12.929794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:12.929794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:12.929794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:12.929794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:12.929794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:12.929794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:12.929794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:12.929794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:12.929794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:12.929794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:12.929794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:12.929794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:12.929794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:12.929794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:12.929794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:12.931050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:12.931050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:12.931050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:12.931050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:12.931050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:12.931050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:12.931050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:12.931050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:12.931050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:12.931050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:12.931050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:12.931050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:12.931050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 253/502] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:43:12 2022] GET /identity/v3/roles/service => generated 85 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:12.931726 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4b501b-efd4-43b9-94c1-00c26d2a5cc3 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:12.994975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 250/503] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:43:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:43:12.996292 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dd9a8a-7450-449b-8ff3-a2a5af4200ec None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:13.026359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:13.055376 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4b501b-efd4-43b9-94c1-00c26d2a5cc3 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:13.097821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa6b539-c0a5-4558-98bb-b070783875a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:13.098549 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa6b539-c0a5-4558-98bb-b070783875a3 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:13.098900 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa6b539-c0a5-4558-98bb-b070783875a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:13.099238 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa6b539-c0a5-4558-98bb-b070783875a3 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:13.099994 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa6b539-c0a5-4558-98bb-b070783875a3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:13.100369 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa6b539-c0a5-4558-98bb-b070783875a3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:13.102213 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa6b539-c0a5-4558-98bb-b070783875a3 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:13.111984 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 254/504] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 18 00:43:12 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:13.113568 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa6b539-c0a5-4558-98bb-b070783875a3 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:13.113746 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:13.139247 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dd9a8a-7450-449b-8ff3-a2a5af4200ec None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:13.180889 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa736feb-4171-4e96-8dcf-01f06c77ee3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:13.181514 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa736feb-4171-4e96-8dcf-01f06c77ee3d None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:13.181795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa736feb-4171-4e96-8dcf-01f06c77ee3d None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:13.182068 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa736feb-4171-4e96-8dcf-01f06c77ee3d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:13.253951 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa736feb-4171-4e96-8dcf-01f06c77ee3d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:13.255468 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa736feb-4171-4e96-8dcf-01f06c77ee3d None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:13.271368 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa6b539-c0a5-4558-98bb-b070783875a3 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:13.329255 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbb9ee3-62f8-4e00-9157-e6b0cbb98610 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:13.330268 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbb9ee3-62f8-4e00-9157-e6b0cbb98610 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:13.330793 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbb9ee3-62f8-4e00-9157-e6b0cbb98610 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:13.331258 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbb9ee3-62f8-4e00-9157-e6b0cbb98610 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:13.332450 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbb9ee3-62f8-4e00-9157-e6b0cbb98610 None admin] RBAC: Adding query filter params (name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:13.332958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbb9ee3-62f8-4e00-9157-e6b0cbb98610 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:13.335563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbb9ee3-62f8-4e00-9157-e6b0cbb98610 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:13.344373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:13.358933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 255/505] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 18 00:43:13 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:13.358933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14406ac-3690-4259-b06c-384961738d3e None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:13.358933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14406ac-3690-4259-b06c-384961738d3e None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:13.358933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14406ac-3690-4259-b06c-384961738d3e None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:13.358933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 256/506] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:13 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:13.358933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14406ac-3690-4259-b06c-384961738d3e None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:13.365558 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:13.372065 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 251/507] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:43:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:43:13.373015 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa736feb-4171-4e96-8dcf-01f06c77ee3d None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:13.482011 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa736feb-4171-4e96-8dcf-01f06c77ee3d None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:13.526973 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14406ac-3690-4259-b06c-384961738d3e None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:13.530109 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad4995f-e914-41a5-b3e5-6960538dc651 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:13.531133 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad4995f-e914-41a5-b3e5-6960538dc651 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:13.531509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad4995f-e914-41a5-b3e5-6960538dc651 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:13.531889 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad4995f-e914-41a5-b3e5-6960538dc651 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:13.547708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad4995f-e914-41a5-b3e5-6960538dc651 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:13.550435 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad4995f-e914-41a5-b3e5-6960538dc651 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:13.570900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-4ad4995f-e914-41a5-b3e5-6960538dc651 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:13.570900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:13.570900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:13.570900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:13.570900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:13.570900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:13.570900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:13.570900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:13.570900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:13.570900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:13.570900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:13.570900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:13.570900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:13.570900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:13.570900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:13.570900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:13.572552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:13.572552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:13.572552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:13.572552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:13.572552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:13.572552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:13.572552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:13.572552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:13.572552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:13.572552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:13.572552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:13.572552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:13.572552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:13.572552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:13.572552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:13.572552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:13.572552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:13.573497 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:13.573497 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:13.573497 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:13.573497 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 252/508] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:13.573727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad4995f-e914-41a5-b3e5-6960538dc651 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:13.586180 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132ecd9f-64b3-421f-995d-04f039ab7ebc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:13.587311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132ecd9f-64b3-421f-995d-04f039ab7ebc None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:13.587766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132ecd9f-64b3-421f-995d-04f039ab7ebc None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:13.588221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132ecd9f-64b3-421f-995d-04f039ab7ebc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:13.608688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-132ecd9f-64b3-421f-995d-04f039ab7ebc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:13.611504 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-132ecd9f-64b3-421f-995d-04f039ab7ebc None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:13.616628 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:13.632269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-132ecd9f-64b3-421f-995d-04f039ab7ebc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:13.632269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:13.632269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:13.632269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:13.632269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:13.632269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:13.632269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:13.632269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:13.632269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:13.632269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:13.632269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:13.632269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:13.632269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:13.632269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:13.632269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:13.632269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:13.634000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:13.634000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:13.634000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:13.634000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:13.634000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:13.634000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:13.634000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:13.634000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:13.634000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:13.634000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:13.634000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:13.634000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:13.634000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:13.634000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:13.634000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:13.634000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:13.634000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:13.635587 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:13.635587 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:13.635587 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:13.638699 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 257/509] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:13.640911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa237c25-1b1f-42dd-9f4a-7f17130c8d63 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:13.641352 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa237c25-1b1f-42dd-9f4a-7f17130c8d63 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:13.641732 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa237c25-1b1f-42dd-9f4a-7f17130c8d63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:13.668718 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-aa237c25-1b1f-42dd-9f4a-7f17130c8d63 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:13.670236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-aa237c25-1b1f-42dd-9f4a-7f17130c8d63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:13.709968 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad4995f-e914-41a5-b3e5-6960538dc651 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:13.732578 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:13.753316 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d20d6ad-0788-43af-982c-e724a997923e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:13.754130 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d20d6ad-0788-43af-982c-e724a997923e None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:13.754285 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 258/510] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:43:13.754613 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d20d6ad-0788-43af-982c-e724a997923e None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:13.755025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d20d6ad-0788-43af-982c-e724a997923e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:13.755687 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa237c25-1b1f-42dd-9f4a-7f17130c8d63 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:13.769390 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d20d6ad-0788-43af-982c-e724a997923e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:13.771788 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d20d6ad-0788-43af-982c-e724a997923e None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:13.790248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-1d20d6ad-0788-43af-982c-e724a997923e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:13.790248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:13.790248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:13.790248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:13.790248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:13.790248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:13.790248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:13.790248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:13.790248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:13.790248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:13.790248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:13.790248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:13.790248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:13.790248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:13.790248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:13.790248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:13.792002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:13.792002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:13.792002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:13.792002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:13.792002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:13.792002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:13.792002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:13.792002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:13.792002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:13.792002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:13.792002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:13.792002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:13.792002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:13.792002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:13.792002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:13.792002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:13.792002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:13.794121 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:13.794121 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:13.794121 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:13.794121 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 253/511] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:13.795888 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d20d6ad-0788-43af-982c-e724a997923e None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:13.922688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:13.940784 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d20d6ad-0788-43af-982c-e724a997923e None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:13.947734 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa237c25-1b1f-42dd-9f4a-7f17130c8d63 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:13.994704 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27af64fb-7204-42d3-af0b-cb2172149ab5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:13.995716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27af64fb-7204-42d3-af0b-cb2172149ab5 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:13.996162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27af64fb-7204-42d3-af0b-cb2172149ab5 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:13.996805 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27af64fb-7204-42d3-af0b-cb2172149ab5 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:13.997248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27af64fb-7204-42d3-af0b-cb2172149ab5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:13.997744 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27af64fb-7204-42d3-af0b-cb2172149ab5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:14.000097 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27af64fb-7204-42d3-af0b-cb2172149ab5 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:14.004137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba057fb-85bf-4b74-b74d-42bb360659aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:14.005148 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba057fb-85bf-4b74-b74d-42bb360659aa None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:14.005587 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba057fb-85bf-4b74-b74d-42bb360659aa None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:14.006025 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba057fb-85bf-4b74-b74d-42bb360659aa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:14.025977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 254/512] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:14.025977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27af64fb-7204-42d3-af0b-cb2172149ab5 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:14.031688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba057fb-85bf-4b74-b74d-42bb360659aa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:14.034234 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba057fb-85bf-4b74-b74d-42bb360659aa None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:14.055796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-3ba057fb-85bf-4b74-b74d-42bb360659aa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:14.055796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:14.055796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:14.055796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:14.055796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:14.055796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:14.055796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:14.055796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:14.055796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:14.055796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:14.055796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:14.055796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:14.055796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:14.055796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:14.055796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:14.055796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:14.057489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:14.057489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:14.057489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:14.057489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:14.057489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:14.057489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:14.057489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:14.057489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:14.057489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:14.057489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:14.057489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:14.057489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:14.057489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:14.057489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:14.057489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:14.057489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:14.057489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:14.060547 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:14.060547 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:14.060547 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:14.060547 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 259/513] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 303 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:14.060547 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba057fb-85bf-4b74-b74d-42bb360659aa None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:14.111448 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:14.196618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27af64fb-7204-42d3-af0b-cb2172149ab5 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:14.218507 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba057fb-85bf-4b74-b74d-42bb360659aa None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:14.249758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c9ab82-2d99-4a22-b9b2-6ce54522325b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:14.250609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c9ab82-2d99-4a22-b9b2-6ce54522325b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:14.251035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c9ab82-2d99-4a22-b9b2-6ce54522325b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:14.251472 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c9ab82-2d99-4a22-b9b2-6ce54522325b None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:14.252536 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c9ab82-2d99-4a22-b9b2-6ce54522325b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:14.253025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c9ab82-2d99-4a22-b9b2-6ce54522325b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:14.255419 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c9ab82-2d99-4a22-b9b2-6ce54522325b None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:14.272202 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3945a3de-e666-4545-994f-c533affcf568 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:14.273132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3945a3de-e666-4545-994f-c533affcf568 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:14.273567 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3945a3de-e666-4545-994f-c533affcf568 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:14.274001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3945a3de-e666-4545-994f-c533affcf568 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:14.274591 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 255/514] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:14.275129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3945a3de-e666-4545-994f-c533affcf568 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:14.275616 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3945a3de-e666-4545-994f-c533affcf568 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:14.275923 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c9ab82-2d99-4a22-b9b2-6ce54522325b None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:14.278057 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3945a3de-e666-4545-994f-c533affcf568 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:14.290405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:14.293620 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 260/515] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:14.294829 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3945a3de-e666-4545-994f-c533affcf568 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:14.427933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c9ab82-2d99-4a22-b9b2-6ce54522325b None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:14.440368 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:14.449950 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3945a3de-e666-4545-994f-c533affcf568 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:14.475080 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10307705-a815-42f1-8e8e-5ea3795d699d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:14.476078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10307705-a815-42f1-8e8e-5ea3795d699d None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:14.476517 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10307705-a815-42f1-8e8e-5ea3795d699d None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:14.476959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10307705-a815-42f1-8e8e-5ea3795d699d None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:14.490005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10307705-a815-42f1-8e8e-5ea3795d699d None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:14.491668 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10307705-a815-42f1-8e8e-5ea3795d699d None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:14.504408 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ee485a-c64f-4fe3-a509-1613812cfd03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:14.505343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ee485a-c64f-4fe3-a509-1613812cfd03 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:14.505784 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ee485a-c64f-4fe3-a509-1613812cfd03 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:14.506222 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ee485a-c64f-4fe3-a509-1613812cfd03 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:14.507333 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ee485a-c64f-4fe3-a509-1613812cfd03 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:14.507833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ee485a-c64f-4fe3-a509-1613812cfd03 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:14.510210 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ee485a-c64f-4fe3-a509-1613812cfd03 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:14.512209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-10307705-a815-42f1-8e8e-5ea3795d699d None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 18 00:43:14.512209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:14.512209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:14.512209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:14.512209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:14.512209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:14.512209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:14.512209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:14.512209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:14.512209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:14.512209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:14.512209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:14.512209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:43:14.512209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:43:14.512209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:43:14.512209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:43:14.513095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:14.513095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:14.513095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:43:14.513095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:14.513095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:43:14.513095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:14.513095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:14.513095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:14.513095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:14.513095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:14.513095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:14.513095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:14.513095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:14.513095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:14.513095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:14.513095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:14.513095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:14.513095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:14.513959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:14.513959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:14.513959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:43:14.513959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:43:14.513959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:43:14.513959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:43:14.513959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:43:14.513959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:43:14.513959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 18 00:43:14.513959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:14.513959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 256/516] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:43:14 2022] GET /identity/v3/users/cinder => generated 84 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:14.515120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10307705-a815-42f1-8e8e-5ea3795d699d None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:14.539149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 261/517] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:14.540483 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ee485a-c64f-4fe3-a509-1613812cfd03 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:14.596863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:14.644440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10307705-a815-42f1-8e8e-5ea3795d699d None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:14.678301 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ee485a-c64f-4fe3-a509-1613812cfd03 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:14.686870 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf3cd36-90d2-4ac2-839e-d4793441b658 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:14.687884 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf3cd36-90d2-4ac2-839e-d4793441b658 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:14.688331 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf3cd36-90d2-4ac2-839e-d4793441b658 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:14.688769 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf3cd36-90d2-4ac2-839e-d4793441b658 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:14.707961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf3cd36-90d2-4ac2-839e-d4793441b658 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:14.710383 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf3cd36-90d2-4ac2-839e-d4793441b658 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:14.730794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cdee4e-a85b-4fb4-bc5d-d5a3a54fa620 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:14.731508 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cdee4e-a85b-4fb4-bc5d-d5a3a54fa620 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:14.731845 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cdee4e-a85b-4fb4-bc5d-d5a3a54fa620 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:14.732186 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cdee4e-a85b-4fb4-bc5d-d5a3a54fa620 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:14.737698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-4cf3cd36-90d2-4ac2-839e-d4793441b658 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:14.737698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:14.737698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:14.737698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:14.737698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:14.737698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:14.737698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:14.737698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:14.737698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:14.737698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:14.737698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:14.737698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:14.737698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:14.737698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:14.737698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:14.737698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:14.738642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:14.738642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:14.738642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:14.738642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:14.738642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:14.738642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:14.738642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:14.738642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:14.738642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:14.738642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:14.738642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:14.738642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:14.738642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:14.738642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:14.738642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:14.738642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:14.738642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:14.739481 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:14.739481 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:14.739481 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:14.739902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 257/518] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:14.740948 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97cdee4e-a85b-4fb4-bc5d-d5a3a54fa620 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:14.741211 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf3cd36-90d2-4ac2-839e-d4793441b658 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:14.743893 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97cdee4e-a85b-4fb4-bc5d-d5a3a54fa620 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:14.758631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-97cdee4e-a85b-4fb4-bc5d-d5a3a54fa620 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:14.758631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:14.758631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:14.758631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:14.758631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:14.758631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:14.758631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:14.758631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:14.758631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:14.758631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:14.758631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:14.758631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:14.758631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:43:14.758631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:43:14.758631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:43:14.760276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:43:14.760276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:14.760276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:14.760276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:14.760276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:14.760276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:14.760276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:14.760276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:14.760276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:14.760276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:14.760276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:14.760276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:14.760276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:14.760276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:14.760276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:14.760276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:14.760276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:14.762933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:43:14.762933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:43:14.762933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:43:14.762933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:43:14.762933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:43:14.762933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:43:14.762933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:14.762933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:14.762933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 262/519] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:14.762933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cdee4e-a85b-4fb4-bc5d-d5a3a54fa620 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:14.780665 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:14.873532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cdee4e-a85b-4fb4-bc5d-d5a3a54fa620 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:14.909952 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf3cd36-90d2-4ac2-839e-d4793441b658 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:14.910180 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea1a94b-0e23-4a2c-87a6-66c221ea5453 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:14.911011 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea1a94b-0e23-4a2c-87a6-66c221ea5453 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:14.911374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea1a94b-0e23-4a2c-87a6-66c221ea5453 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:14.911873 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea1a94b-0e23-4a2c-87a6-66c221ea5453 None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:14.912798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea1a94b-0e23-4a2c-87a6-66c221ea5453 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:14.913161 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea1a94b-0e23-4a2c-87a6-66c221ea5453 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:14.914831 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea1a94b-0e23-4a2c-87a6-66c221ea5453 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:14.937946 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 263/520] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Sep 18 00:43:14 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:14.938829 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea1a94b-0e23-4a2c-87a6-66c221ea5453 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:14.947216 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:14.966160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b41894-c722-40db-97a9-22cfe63cf52d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:14.966753 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b41894-c722-40db-97a9-22cfe63cf52d None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:14.967029 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b41894-c722-40db-97a9-22cfe63cf52d None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:14.967298 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b41894-c722-40db-97a9-22cfe63cf52d None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:14.975408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b41894-c722-40db-97a9-22cfe63cf52d None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:14.976871 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b41894-c722-40db-97a9-22cfe63cf52d None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:14.989069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-80b41894-c722-40db-97a9-22cfe63cf52d None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 18 00:43:14.989069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:14.989069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:14.989069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:14.989069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:14.989069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:14.989069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:14.989069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:14.989069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:14.989069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:14.989069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:14.989069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:14.989069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:43:14.989069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:43:14.989069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:43:14.989069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:43:14.990649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:14.990649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:14.990649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:43:14.990649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:14.990649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:43:14.990649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:14.990649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:14.990649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:14.990649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:14.990649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:14.990649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:14.990649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:14.990649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:14.990649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:14.990649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:14.990649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:14.990649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:14.990649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:14.992344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:14.992344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:14.992344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:43:14.992344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:43:14.992344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:43:14.992344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:43:14.992344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:43:14.992344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:43:14.992344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 18 00:43:14.992344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:14.992344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 258/521] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:43:14 2022] GET /identity/v3/users/neutron => generated 85 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:14.992344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b41894-c722-40db-97a9-22cfe63cf52d None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:15.036164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:15.063766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea1a94b-0e23-4a2c-87a6-66c221ea5453 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:15.099582 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e01e184-5a8d-4c05-8c0f-b098fe596bf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:15.100218 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e01e184-5a8d-4c05-8c0f-b098fe596bf1 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:15.100504 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e01e184-5a8d-4c05-8c0f-b098fe596bf1 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:15.100764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e01e184-5a8d-4c05-8c0f-b098fe596bf1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:15.107994 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e01e184-5a8d-4c05-8c0f-b098fe596bf1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:15.109463 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e01e184-5a8d-4c05-8c0f-b098fe596bf1 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:15.121097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-0e01e184-5a8d-4c05-8c0f-b098fe596bf1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:15.121097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:15.121097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:15.121097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:15.121097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:15.121097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:15.121097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:15.121097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:15.121097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:15.121097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:15.121097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:15.121097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:15.121097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:43:15.121097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:43:15.121097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:43:15.129134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:43:15.129134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:15.129134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:15.129134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:15.129134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:15.129134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:15.129134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:15.129134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:15.129134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:15.129134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:15.129134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:15.129134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:15.129134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:15.129134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:15.129134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:15.129134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:15.129134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:15.130446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:43:15.130446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:43:15.130446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:43:15.130446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:43:15.130446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:43:15.130446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:43:15.130446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:15.130446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:15.130446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 264/522] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:15.130446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e01e184-5a8d-4c05-8c0f-b098fe596bf1 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:15.139004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b41894-c722-40db-97a9-22cfe63cf52d None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:15.175739 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa2b6b7-ae4e-40fd-a447-74a543118c79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:15.176354 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa2b6b7-ae4e-40fd-a447-74a543118c79 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:15.176636 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa2b6b7-ae4e-40fd-a447-74a543118c79 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:15.176905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa2b6b7-ae4e-40fd-a447-74a543118c79 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:15.177550 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa2b6b7-ae4e-40fd-a447-74a543118c79 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:15.177850 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa2b6b7-ae4e-40fd-a447-74a543118c79 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:15.179266 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa2b6b7-ae4e-40fd-a447-74a543118c79 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:15.192064 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 259/523] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:15.192901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa2b6b7-ae4e-40fd-a447-74a543118c79 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:15.256640 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:15.274010 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e01e184-5a8d-4c05-8c0f-b098fe596bf1 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:15.320304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d892407-f7d7-451c-b9a1-45cafdf08c59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:15.321102 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d892407-f7d7-451c-b9a1-45cafdf08c59 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:15.321381 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d892407-f7d7-451c-b9a1-45cafdf08c59 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:15.321656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d892407-f7d7-451c-b9a1-45cafdf08c59 None admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:15.322371 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d892407-f7d7-451c-b9a1-45cafdf08c59 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:15.322707 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d892407-f7d7-451c-b9a1-45cafdf08c59 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:15.324218 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d892407-f7d7-451c-b9a1-45cafdf08c59 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:15.326242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:15.332940 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa2b6b7-ae4e-40fd-a447-74a543118c79 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:15.346941 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 265/524] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:43:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:15.347912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d892407-f7d7-451c-b9a1-45cafdf08c59 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:15.372434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d68df8-f2ac-458c-90fb-5fc52e65038f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:15.373056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d68df8-f2ac-458c-90fb-5fc52e65038f None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:15.373339 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d68df8-f2ac-458c-90fb-5fc52e65038f None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:15.373614 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d68df8-f2ac-458c-90fb-5fc52e65038f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:15.446044 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d68df8-f2ac-458c-90fb-5fc52e65038f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:15.447803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d68df8-f2ac-458c-90fb-5fc52e65038f None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:15.475028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d892407-f7d7-451c-b9a1-45cafdf08c59 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:15.493669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:15.525811 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f11724-011e-4975-9d99-3461fd5d46de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:15.527427 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f11724-011e-4975-9d99-3461fd5d46de None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:15.527884 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f11724-011e-4975-9d99-3461fd5d46de None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:15.528230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f11724-011e-4975-9d99-3461fd5d46de None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:15.529216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f11724-011e-4975-9d99-3461fd5d46de None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:15.529712 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f11724-011e-4975-9d99-3461fd5d46de None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:15.532153 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f11724-011e-4975-9d99-3461fd5d46de None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:15.561482 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 266/525] 10.222.0.42 () {56 vars in 1139 bytes} [Sun Sep 18 00:43:15 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:15.562723 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f11724-011e-4975-9d99-3461fd5d46de None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:15.581710 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 260/526] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:43:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:43:15.583050 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d68df8-f2ac-458c-90fb-5fc52e65038f None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:15.631530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:15.690592 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f11724-011e-4975-9d99-3461fd5d46de None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:15.707643 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d68df8-f2ac-458c-90fb-5fc52e65038f None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:15.727894 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c579707-951c-4133-b33f-7ae3605216c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:15.728507 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c579707-951c-4133-b33f-7ae3605216c9 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:15.728784 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c579707-951c-4133-b33f-7ae3605216c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:15.729054 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c579707-951c-4133-b33f-7ae3605216c9 None admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:15.729752 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c579707-951c-4133-b33f-7ae3605216c9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:15.730611 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c579707-951c-4133-b33f-7ae3605216c9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:15.732087 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c579707-951c-4133-b33f-7ae3605216c9 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:15.746055 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 267/527] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:43:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:15.747283 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c579707-951c-4133-b33f-7ae3605216c9 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:15.753941 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ede5da-1d4a-4949-9630-613da0c10a72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:15.754755 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ede5da-1d4a-4949-9630-613da0c10a72 None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:15.755152 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ede5da-1d4a-4949-9630-613da0c10a72 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:15.755544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ede5da-1d4a-4949-9630-613da0c10a72 None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:15.756332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ede5da-1d4a-4949-9630-613da0c10a72 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:15.758037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ede5da-1d4a-4949-9630-613da0c10a72 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:15.772126 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:15.790749 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 261/528] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 18 00:43:15 2022] POST /identity/v3/users => generated 247 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:43:15.792075 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ede5da-1d4a-4949-9630-613da0c10a72 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:15.921465 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c579707-951c-4133-b33f-7ae3605216c9 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:15.943659 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:15.950122 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ede5da-1d4a-4949-9630-613da0c10a72 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:15.984092 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797adcd1-a8ae-4f30-8741-9e094fdca250 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:15.985055 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797adcd1-a8ae-4f30-8741-9e094fdca250 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:15.985556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797adcd1-a8ae-4f30-8741-9e094fdca250 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:15.986047 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797adcd1-a8ae-4f30-8741-9e094fdca250 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:15.987545 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797adcd1-a8ae-4f30-8741-9e094fdca250 None admin] RBAC: Adding query filter params (role.id=81523e544c7e4993af4d1e06ce3754bc, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=d7fe9d21099e4c4891c28687bb673214) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:15.988107 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797adcd1-a8ae-4f30-8741-9e094fdca250 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:15.990627 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797adcd1-a8ae-4f30-8741-9e094fdca250 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:15.995266 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57dcf34f-f157-4bac-96d8-696f48d88c27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:15.995961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dcf34f-f157-4bac-96d8-696f48d88c27 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:15.996259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dcf34f-f157-4bac-96d8-696f48d88c27 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:15.996521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dcf34f-f157-4bac-96d8-696f48d88c27 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:16.002342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 268/529] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:43:15 2022] GET /identity/v3/role_assignments?user.id=d7fe9d21099e4c4891c28687bb673214&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=81523e544c7e4993af4d1e06ce3754bc => generated 259 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:16.003325 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797adcd1-a8ae-4f30-8741-9e094fdca250 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:16.097243 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:16.110791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57dcf34f-f157-4bac-96d8-696f48d88c27 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:16.110791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57dcf34f-f157-4bac-96d8-696f48d88c27 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:16.202137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797adcd1-a8ae-4f30-8741-9e094fdca250 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:16.261881 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9df07a-84b6-4f40-bb23-7c1089daa617 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:16.263304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9df07a-84b6-4f40-bb23-7c1089daa617 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:16.263771 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9df07a-84b6-4f40-bb23-7c1089daa617 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:16.264214 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9df07a-84b6-4f40-bb23-7c1089daa617 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:16.271079 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 262/530] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:43:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 480 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:43:16.277542 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57dcf34f-f157-4bac-96d8-696f48d88c27 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:16.279464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9df07a-84b6-4f40-bb23-7c1089daa617 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:16.283202 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9df07a-84b6-4f40-bb23-7c1089daa617 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:16.305978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-1e9df07a-84b6-4f40-bb23-7c1089daa617 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:16.305978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:16.305978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:16.305978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:16.305978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:16.305978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:16.305978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:16.305978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:16.305978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:16.305978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:16.305978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:16.305978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:16.305978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:16.305978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:16.305978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:16.305978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:16.308744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:16.308744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:16.308744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:16.308744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:16.308744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:16.308744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:16.308744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:16.308744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:16.308744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:16.308744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:16.308744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:16.308744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:16.308744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:16.308744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:16.308744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:16.308744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:16.308744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:16.310384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:16.310384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:16.310384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:16.313670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 269/531] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 306 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:16.315032 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9df07a-84b6-4f40-bb23-7c1089daa617 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:16.363862 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:16.440189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:16.450142 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57dcf34f-f157-4bac-96d8-696f48d88c27 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:16.503492 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9df07a-84b6-4f40-bb23-7c1089daa617 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:16.525467 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e30960-1b0b-407c-90eb-c78075281365 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:16.526645 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e30960-1b0b-407c-90eb-c78075281365 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:16.527286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e30960-1b0b-407c-90eb-c78075281365 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:16.527956 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e30960-1b0b-407c-90eb-c78075281365 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:16.529524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e30960-1b0b-407c-90eb-c78075281365 None admin] RBAC: Adding query filter params (role.id=81523e544c7e4993af4d1e06ce3754bc, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=a59d73fd45a54651930d52300704fdb0) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:16.530274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e30960-1b0b-407c-90eb-c78075281365 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:16.533262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e30960-1b0b-407c-90eb-c78075281365 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:16.550666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 263/532] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:43:16 2022] GET /identity/v3/role_assignments?user.id=a59d73fd45a54651930d52300704fdb0&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=81523e544c7e4993af4d1e06ce3754bc => generated 259 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:16.552229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e30960-1b0b-407c-90eb-c78075281365 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:16.586035 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52a938d-fecb-455a-ad11-1b4cc3b2ef09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:16.587100 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52a938d-fecb-455a-ad11-1b4cc3b2ef09 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:16.587631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52a938d-fecb-455a-ad11-1b4cc3b2ef09 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:16.588103 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52a938d-fecb-455a-ad11-1b4cc3b2ef09 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:16.615587 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52a938d-fecb-455a-ad11-1b4cc3b2ef09 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:16.617405 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52a938d-fecb-455a-ad11-1b4cc3b2ef09 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:16.655386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-b52a938d-fecb-455a-ad11-1b4cc3b2ef09 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:16.655386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:16.655386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:16.655386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:16.655386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:16.655386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:16.655386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:16.655386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:16.655386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:16.655386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:16.655386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:16.655386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:16.655386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:16.655386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:16.655386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:16.655386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:16.656389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:16.656389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:16.656389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:16.656389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:16.656389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:16.656389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:16.656389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:16.656389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:16.656389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:16.656389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:16.656389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:16.656389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:16.656389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:16.656389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:16.656389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:16.656389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:16.656389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:16.657222 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:16.657222 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:16.657222 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:16.659265 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 270/533] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 345 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:16.665361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52a938d-fecb-455a-ad11-1b4cc3b2ef09 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:16.790846 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:16.814126 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52a938d-fecb-455a-ad11-1b4cc3b2ef09 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:16.815922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e30960-1b0b-407c-90eb-c78075281365 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:16.857128 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d458e10-0d0d-4886-a1c3-ace07dbef99c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:16.857747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d458e10-0d0d-4886-a1c3-ace07dbef99c None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:16.858021 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d458e10-0d0d-4886-a1c3-ace07dbef99c None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:16.858506 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d458e10-0d0d-4886-a1c3-ace07dbef99c None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:16.859181 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d458e10-0d0d-4886-a1c3-ace07dbef99c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:16.859480 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d458e10-0d0d-4886-a1c3-ace07dbef99c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:16.860931 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d458e10-0d0d-4886-a1c3-ace07dbef99c None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:16.875878 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 271/534] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:16.885220 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d458e10-0d0d-4886-a1c3-ace07dbef99c None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:16.890145 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef331d9-63d2-4753-884f-a24b6c85eb4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:16.890846 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef331d9-63d2-4753-884f-a24b6c85eb4e None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:16.891247 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef331d9-63d2-4753-884f-a24b6c85eb4e None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:16.891525 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef331d9-63d2-4753-884f-a24b6c85eb4e None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:16.892236 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef331d9-63d2-4753-884f-a24b6c85eb4e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:16.892543 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef331d9-63d2-4753-884f-a24b6c85eb4e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:16.894015 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef331d9-63d2-4753-884f-a24b6c85eb4e None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:16.909070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 264/535] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 357 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:16.917454 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef331d9-63d2-4753-884f-a24b6c85eb4e None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:16.967814 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:17.072987 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef331d9-63d2-4753-884f-a24b6c85eb4e None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:17.109214 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53eb355f-f350-493f-8dbc-cc06a84150b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:17.109868 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53eb355f-f350-493f-8dbc-cc06a84150b6 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:17.110148 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53eb355f-f350-493f-8dbc-cc06a84150b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:17.110422 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53eb355f-f350-493f-8dbc-cc06a84150b6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:17.118048 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53eb355f-f350-493f-8dbc-cc06a84150b6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:17.119574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53eb355f-f350-493f-8dbc-cc06a84150b6 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:17.133141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-53eb355f-f350-493f-8dbc-cc06a84150b6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:17.133141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:17.133141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:17.133141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:17.133141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:17.133141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:17.133141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:17.133141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:17.133141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:17.133141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:17.133141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:17.133141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:17.133141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:43:17.133141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:43:17.133141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:43:17.134257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:43:17.134257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:17.134257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:17.134257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:17.134257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:17.134257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:17.134257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:17.134257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:17.134257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:17.134257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:17.134257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:17.134257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:17.134257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:17.134257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:17.134257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:17.134257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:17.134257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:17.135153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:43:17.135153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:43:17.135153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:43:17.135153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:43:17.135153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:43:17.135153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:43:17.135153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:17.135153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:17.138617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 265/536] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:17.140703 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53eb355f-f350-493f-8dbc-cc06a84150b6 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:17.188511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d458e10-0d0d-4886-a1c3-ace07dbef99c None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:17.232452 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:17.258926 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53eb355f-f350-493f-8dbc-cc06a84150b6 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:17.270128 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472f1f43-3d65-4fb3-8ceb-5f1568d7fdca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:17.270961 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472f1f43-3d65-4fb3-8ceb-5f1568d7fdca None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:17.271443 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472f1f43-3d65-4fb3-8ceb-5f1568d7fdca None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:17.271821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472f1f43-3d65-4fb3-8ceb-5f1568d7fdca None admin] PATH_INFO: `/v3/projects/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:17.281260 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472f1f43-3d65-4fb3-8ceb-5f1568d7fdca None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:17.282949 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472f1f43-3d65-4fb3-8ceb-5f1568d7fdca None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:17.286174 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:17.297215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-472f1f43-3d65-4fb3-8ceb-5f1568d7fdca None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:17.297215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:17.297215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:17.297215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:17.297215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:17.297215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:17.297215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:17.297215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:17.297215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:17.297215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:17.297215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:17.297215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:17.297215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:43:17.297215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:43:17.297215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:43:17.298127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:43:17.298127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:17.298127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:17.298127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:17.298127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:17.298127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:17.298127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:17.298127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:17.298127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:17.298127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:17.298127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:17.298127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:17.298127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:17.298127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:17.298127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:17.298127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:17.298127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:17.299015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:43:17.299015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:43:17.299015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:43:17.299015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:43:17.299015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:43:17.299015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:43:17.299015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:17.299015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:17.305021 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 272/537] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 420 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:17.306420 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472f1f43-3d65-4fb3-8ceb-5f1568d7fdca None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:17.331627 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60125fea-917b-464f-8323-3f5b73b6a279 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:17.332309 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60125fea-917b-464f-8323-3f5b73b6a279 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:17.332569 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60125fea-917b-464f-8323-3f5b73b6a279 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:17.332844 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60125fea-917b-464f-8323-3f5b73b6a279 None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:17.333551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60125fea-917b-464f-8323-3f5b73b6a279 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:17.334725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60125fea-917b-464f-8323-3f5b73b6a279 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:17.336203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60125fea-917b-464f-8323-3f5b73b6a279 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:17.382052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 266/538] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:43:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:43:17.385444 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60125fea-917b-464f-8323-3f5b73b6a279 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:17.526351 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60125fea-917b-464f-8323-3f5b73b6a279 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:17.560529 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472f1f43-3d65-4fb3-8ceb-5f1568d7fdca None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:17.564585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76fc8b8-1ade-4b99-9c5c-830ac3ce77e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:17.565303 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76fc8b8-1ade-4b99-9c5c-830ac3ce77e6 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:17.565594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76fc8b8-1ade-4b99-9c5c-830ac3ce77e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:17.565874 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76fc8b8-1ade-4b99-9c5c-830ac3ce77e6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:17.566666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76fc8b8-1ade-4b99-9c5c-830ac3ce77e6 None admin] RBAC: Adding query filter params (role.id=81523e544c7e4993af4d1e06ce3754bc, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=c4c3eaee0a8f4de0b79841a95b35091f) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:17.566984 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76fc8b8-1ade-4b99-9c5c-830ac3ce77e6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:17.568450 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76fc8b8-1ade-4b99-9c5c-830ac3ce77e6 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:17.577341 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 267/539] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:43:17 2022] GET /identity/v3/role_assignments?user.id=c4c3eaee0a8f4de0b79841a95b35091f&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=81523e544c7e4993af4d1e06ce3754bc => generated 259 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:17.623658 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815c18fe-c47c-4511-9fb5-033a77394e90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:17.624302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815c18fe-c47c-4511-9fb5-033a77394e90 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:17.624589 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815c18fe-c47c-4511-9fb5-033a77394e90 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:17.624869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815c18fe-c47c-4511-9fb5-033a77394e90 None admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:17.625601 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815c18fe-c47c-4511-9fb5-033a77394e90 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:17.625916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815c18fe-c47c-4511-9fb5-033a77394e90 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:17.632133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815c18fe-c47c-4511-9fb5-033a77394e90 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:17.653141 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 273/540] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:43:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 347 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:43:17.659294 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76fc8b8-1ade-4b99-9c5c-830ac3ce77e6 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:17.663570 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:17.799855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76fc8b8-1ade-4b99-9c5c-830ac3ce77e6 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:17.839065 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98820610-5556-4e6d-bf03-1324e4d4b4ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:17.840228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98820610-5556-4e6d-bf03-1324e4d4b4ea None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:17.840513 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98820610-5556-4e6d-bf03-1324e4d4b4ea None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:17.840787 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98820610-5556-4e6d-bf03-1324e4d4b4ea None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:17.841530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98820610-5556-4e6d-bf03-1324e4d4b4ea None admin] RBAC: Adding query filter params (role.id=81523e544c7e4993af4d1e06ce3754bc, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=9b0510812b60454586a781e6399e77a4) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:17.841841 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98820610-5556-4e6d-bf03-1324e4d4b4ea None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:17.843324 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98820610-5556-4e6d-bf03-1324e4d4b4ea None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:17.849816 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 268/541] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:43:17 2022] GET /identity/v3/role_assignments?user.id=9b0510812b60454586a781e6399e77a4&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=81523e544c7e4993af4d1e06ce3754bc => generated 259 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:18.639949 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01726654-cb64-49b9-9330-855effe69c13 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:18.640612 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01726654-cb64-49b9-9330-855effe69c13 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:18.640612 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01726654-cb64-49b9-9330-855effe69c13 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:18.642136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 274/542] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:18.648151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738f7fb0-3f02-4430-97d7-f6c64c5e37b0 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:18.648877 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738f7fb0-3f02-4430-97d7-f6c64c5e37b0 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:18.648877 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738f7fb0-3f02-4430-97d7-f6c64c5e37b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:18.668739 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-738f7fb0-3f02-4430-97d7-f6c64c5e37b0 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:18.669962 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-738f7fb0-3f02-4430-97d7-f6c64c5e37b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:18.767605 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacbb2a7-c64f-4648-b7d3-717452b45677 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:18.768533 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacbb2a7-c64f-4648-b7d3-717452b45677 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:18.769267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacbb2a7-c64f-4648-b7d3-717452b45677 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:18.772341 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 275/543] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:18 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:18.781899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26644635-f9eb-4b17-b719-146a4364ce06 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:18.782853 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26644635-f9eb-4b17-b719-146a4364ce06 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:18.783601 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26644635-f9eb-4b17-b719-146a4364ce06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:18.789407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:18.801007 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 269/544] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:43:18.836236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-26644635-f9eb-4b17-b719-146a4364ce06 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:18.838328 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-26644635-f9eb-4b17-b719-146a4364ce06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:18.938531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:18.958796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 276/545] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:43:19.147676 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738f7fb0-3f02-4430-97d7-f6c64c5e37b0 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:19.256569 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738f7fb0-3f02-4430-97d7-f6c64c5e37b0 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:19.304000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfe79d1-34d1-4b75-822d-0e09de989b1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:19.304666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfe79d1-34d1-4b75-822d-0e09de989b1b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:19.304955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfe79d1-34d1-4b75-822d-0e09de989b1b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:19.305224 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfe79d1-34d1-4b75-822d-0e09de989b1b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:19.326151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfe79d1-34d1-4b75-822d-0e09de989b1b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:19.339000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-1dfe79d1-34d1-4b75-822d-0e09de989b1b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:19.339000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:19.339000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:19.339000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:19.339000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:19.339000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:19.339000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:19.339000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:19.339000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:19.339000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:19.339000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:19.339000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:19.339000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:19.339000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:19.339000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:19.339000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:19.340764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:19.340764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:19.340764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:19.340764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:19.340764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:19.340764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:19.340764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:19.340764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:19.340764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:19.340764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:19.340764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:19.340764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:19.340764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:19.340764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:19.340764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:19.340764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:19.341557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:19.341557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:19.341557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:19.341557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:19.341557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:19.341557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:19.341557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:19.341557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:19.341557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:19.341557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:19.341557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:19.341557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:19.344597 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 270/546] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:43:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:19.351600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26644635-f9eb-4b17-b719-146a4364ce06 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:19.487804 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26644635-f9eb-4b17-b719-146a4364ce06 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:19.535432 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078daa7b-a37c-4003-b226-c195cdfe792c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:19.542704 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078daa7b-a37c-4003-b226-c195cdfe792c None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:19.542704 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078daa7b-a37c-4003-b226-c195cdfe792c None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:19.542704 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078daa7b-a37c-4003-b226-c195cdfe792c None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:19.542704 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078daa7b-a37c-4003-b226-c195cdfe792c None admin] RBAC: Adding query filter params (name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:19.542704 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078daa7b-a37c-4003-b226-c195cdfe792c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:19.542704 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078daa7b-a37c-4003-b226-c195cdfe792c None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:19.549359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 277/547] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 18 00:43:19 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:19.555610 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfe79d1-34d1-4b75-822d-0e09de989b1b None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:19.715490 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078daa7b-a37c-4003-b226-c195cdfe792c None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:19.719278 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:19.821790 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:19.825855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfe79d1-34d1-4b75-822d-0e09de989b1b None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:19.909681 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e112ff0-9368-4d70-9479-e12c15c50ddb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:19.911133 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e112ff0-9368-4d70-9479-e12c15c50ddb None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:19.914251 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e112ff0-9368-4d70-9479-e12c15c50ddb None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:19.914952 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e112ff0-9368-4d70-9479-e12c15c50ddb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:19.934743 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078daa7b-a37c-4003-b226-c195cdfe792c None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:19.939754 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e112ff0-9368-4d70-9479-e12c15c50ddb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:19.941648 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e112ff0-9368-4d70-9479-e12c15c50ddb None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:19.964092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-4e112ff0-9368-4d70-9479-e12c15c50ddb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:19.964092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:19.964092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:19.964092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:19.964092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:19.964092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:19.964092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:19.964092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:19.964092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:19.964092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:19.964092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:19.964092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:19.964092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:19.964092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:19.964092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:19.964092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:19.965624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:19.965624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:19.965624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:19.965624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:19.965624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:19.965624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:19.965624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:19.965624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:19.965624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:19.965624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:19.965624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:19.965624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:19.965624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:19.965624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:19.965624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:19.965624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:19.965624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:19.967463 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:19.967463 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:19.967463 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:19.970694 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 271/548] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 416 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:19.977022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e112ff0-9368-4d70-9479-e12c15c50ddb None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:20.018447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a08a7d5-9360-4afd-908e-456ba8c1c1f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:20.019129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a08a7d5-9360-4afd-908e-456ba8c1c1f9 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:20.019406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a08a7d5-9360-4afd-908e-456ba8c1c1f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:20.019675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a08a7d5-9360-4afd-908e-456ba8c1c1f9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:20.050971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a08a7d5-9360-4afd-908e-456ba8c1c1f9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:20.059916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-7a08a7d5-9360-4afd-908e-456ba8c1c1f9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:20.059916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:20.059916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:20.059916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:20.059916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:20.059916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:20.059916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:20.059916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:20.059916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:20.059916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:20.059916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:20.059916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:20.059916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:20.059916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:20.059916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:20.059916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:20.061069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:20.061069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:20.061069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:20.061069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:20.061069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:20.061069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:20.061069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:20.061069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:20.061069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:20.061069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:20.061069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:20.061069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:20.061069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:20.061069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:20.061069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:20.061069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:20.061879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:20.061879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:20.061879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:20.061879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:20.061879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:20.061879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:20.061879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:20.061879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:20.061879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:20.061879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:20.061879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:20.061879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:20.064701 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 278/549] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:43:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 351 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:43:20.069268 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a08a7d5-9360-4afd-908e-456ba8c1c1f9 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:20.149878 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e112ff0-9368-4d70-9479-e12c15c50ddb None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:20.166366 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:20.197091 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4818338e-5c78-4715-b246-ebb5c14d9aa8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:20.197870 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4818338e-5c78-4715-b246-ebb5c14d9aa8 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:20.198145 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4818338e-5c78-4715-b246-ebb5c14d9aa8 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:20.198416 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4818338e-5c78-4715-b246-ebb5c14d9aa8 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:20.199561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4818338e-5c78-4715-b246-ebb5c14d9aa8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:20.199862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4818338e-5c78-4715-b246-ebb5c14d9aa8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:20.201574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4818338e-5c78-4715-b246-ebb5c14d9aa8 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:20.221111 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 272/550] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:20.222668 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405b62b1-dc94-472a-a979-1ba3c1ca3f4d None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:20.222953 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405b62b1-dc94-472a-a979-1ba3c1ca3f4d None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:20.223225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405b62b1-dc94-472a-a979-1ba3c1ca3f4d None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:20.225665 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 273/551] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:20.232840 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405b62b1-dc94-472a-a979-1ba3c1ca3f4d None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:20.258389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a08a7d5-9360-4afd-908e-456ba8c1c1f9 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:20.275496 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:20.305342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93a3f24-b5f8-428c-8cc9-7472aed0c10d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:20.306300 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93a3f24-b5f8-428c-8cc9-7472aed0c10d None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:20.306781 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93a3f24-b5f8-428c-8cc9-7472aed0c10d None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:20.307282 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93a3f24-b5f8-428c-8cc9-7472aed0c10d None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:20.308630 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93a3f24-b5f8-428c-8cc9-7472aed0c10d None admin] RBAC: Adding query filter params (name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:20.309137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93a3f24-b5f8-428c-8cc9-7472aed0c10d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:20.311625 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93a3f24-b5f8-428c-8cc9-7472aed0c10d None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:20.327829 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 279/552] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 18 00:43:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:20.330158 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec04296-21fe-4e1c-a568-b2424510c7d4 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:20.330661 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec04296-21fe-4e1c-a568-b2424510c7d4 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:20.331100 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec04296-21fe-4e1c-a568-b2424510c7d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:20.363342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-fec04296-21fe-4e1c-a568-b2424510c7d4 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:20.365354 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-fec04296-21fe-4e1c-a568-b2424510c7d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:20.417067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405b62b1-dc94-472a-a979-1ba3c1ca3f4d None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:20.467502 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 280/553] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:43:20.470498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec04296-21fe-4e1c-a568-b2424510c7d4 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:20.470630 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a263a7c6-cb7a-42b8-b7ad-6bcd9747b0e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:20.470630 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a263a7c6-cb7a-42b8-b7ad-6bcd9747b0e5 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:20.470630 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a263a7c6-cb7a-42b8-b7ad-6bcd9747b0e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:20.470630 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a263a7c6-cb7a-42b8-b7ad-6bcd9747b0e5 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:20.481201 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a263a7c6-cb7a-42b8-b7ad-6bcd9747b0e5 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:20.482744 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a263a7c6-cb7a-42b8-b7ad-6bcd9747b0e5 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:20.499052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-a263a7c6-cb7a-42b8-b7ad-6bcd9747b0e5 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 18 00:43:20.499052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:20.499052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:20.499052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:20.499052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:20.499052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:20.499052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:20.499052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:20.499052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:20.499052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:20.499052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:20.499052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:20.499052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:43:20.499052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:43:20.499052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:43:20.499052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:43:20.500086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:20.500086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:20.500086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:43:20.500086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:20.500086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:43:20.500086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:20.500086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:20.500086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:20.500086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:20.500086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:20.500086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:20.500086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:20.500086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:20.500086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:20.500086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:20.500086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:20.500086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:20.500086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:20.501085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:20.501085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:20.501085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:43:20.501085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:43:20.501085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:43:20.501085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:43:20.501085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:43:20.501085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:43:20.501085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 18 00:43:20.501085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:20.501085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 274/554] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:43:20 2022] GET /identity/v3/users/swift => generated 83 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:20.502167 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fcdbc3-fb20-4eb1-a8b6-0ad0b855e23b None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:20.502167 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fcdbc3-fb20-4eb1-a8b6-0ad0b855e23b None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:20.502400 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fcdbc3-fb20-4eb1-a8b6-0ad0b855e23b None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:20.514057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 275/555] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:20.514057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bb1c4d-7a56-4c72-9e0c-49a86eb10fbf None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:20.514057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bb1c4d-7a56-4c72-9e0c-49a86eb10fbf None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:20.514057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bb1c4d-7a56-4c72-9e0c-49a86eb10fbf None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:20.514057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 276/556] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:20.514057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea22dd0-f8a8-4317-96db-7ba686028d19 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:20.514057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea22dd0-f8a8-4317-96db-7ba686028d19 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:20.515080 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea22dd0-f8a8-4317-96db-7ba686028d19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:20.537808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-6ea22dd0-f8a8-4317-96db-7ba686028d19 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:20.539137 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-6ea22dd0-f8a8-4317-96db-7ba686028d19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:20.582227 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:20.645790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 277/557] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:43:20.647312 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea22dd0-f8a8-4317-96db-7ba686028d19 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:20.660438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec04296-21fe-4e1c-a568-b2424510c7d4 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:20.664795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:20.734692 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75602f3f-3d34-4efd-9ecc-3d6ff5b66c36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:20.735651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75602f3f-3d34-4efd-9ecc-3d6ff5b66c36 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:20.736076 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75602f3f-3d34-4efd-9ecc-3d6ff5b66c36 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:20.736503 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75602f3f-3d34-4efd-9ecc-3d6ff5b66c36 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:20.762709 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75602f3f-3d34-4efd-9ecc-3d6ff5b66c36 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:20.765537 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75602f3f-3d34-4efd-9ecc-3d6ff5b66c36 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:20.795394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-75602f3f-3d34-4efd-9ecc-3d6ff5b66c36 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:20.795394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:20.795394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:20.795394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:20.795394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:20.795394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:20.795394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:20.795394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:20.795394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:20.795394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:20.795394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:20.795394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:20.795394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:20.795394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:20.795394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:20.795394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:20.796458 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:20.796458 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:20.796458 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:20.796458 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:20.796458 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:20.796458 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:20.796458 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:20.796458 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:20.796458 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:20.796458 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:20.796458 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:20.796458 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:20.796458 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:20.796458 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:20.796458 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:20.796458 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:20.796458 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:20.797524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:20.797524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:20.797524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:20.805513 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 281/558] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 337 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:20.807919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1e437a-53e2-4933-9119-1445b5432b24 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:20.808810 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1e437a-53e2-4933-9119-1445b5432b24 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:20.809255 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1e437a-53e2-4933-9119-1445b5432b24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:20.816577 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea22dd0-f8a8-4317-96db-7ba686028d19 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:20.835029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-7c1e437a-53e2-4933-9119-1445b5432b24 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:20.836276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-7c1e437a-53e2-4933-9119-1445b5432b24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:20.877555 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d433a9b-e20b-4581-92b5-82f4b922cac3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:20.878957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d433a9b-e20b-4581-92b5-82f4b922cac3 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:20.879421 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d433a9b-e20b-4581-92b5-82f4b922cac3 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:20.879832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d433a9b-e20b-4581-92b5-82f4b922cac3 None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:20.880920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d433a9b-e20b-4581-92b5-82f4b922cac3 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:20.881418 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d433a9b-e20b-4581-92b5-82f4b922cac3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:20.883340 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d433a9b-e20b-4581-92b5-82f4b922cac3 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:20.934519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 282/559] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:43:20.934519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1e437a-53e2-4933-9119-1445b5432b24 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:20.944854 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 278/560] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Sep 18 00:43:20 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:20.946257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d433a9b-e20b-4581-92b5-82f4b922cac3 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:20.974304 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:21.069542 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:21.081885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d433a9b-e20b-4581-92b5-82f4b922cac3 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:21.094374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1e437a-53e2-4933-9119-1445b5432b24 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:21.119014 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b7e6a9-450b-452b-8b6d-48ff9301ee1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:21.119726 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b7e6a9-450b-452b-8b6d-48ff9301ee1d None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:21.120011 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b7e6a9-450b-452b-8b6d-48ff9301ee1d None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:21.120284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b7e6a9-450b-452b-8b6d-48ff9301ee1d None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:21.120996 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b7e6a9-450b-452b-8b6d-48ff9301ee1d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:21.121300 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b7e6a9-450b-452b-8b6d-48ff9301ee1d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:21.122729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b7e6a9-450b-452b-8b6d-48ff9301ee1d None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:21.135791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 279/561] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:21.137166 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b7e6a9-450b-452b-8b6d-48ff9301ee1d None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:21.160601 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cb7a6e-f92e-45bd-84fd-a54422ce803b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:21.161523 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cb7a6e-f92e-45bd-84fd-a54422ce803b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:21.161959 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cb7a6e-f92e-45bd-84fd-a54422ce803b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:21.162386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cb7a6e-f92e-45bd-84fd-a54422ce803b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:21.175349 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7cb7a6e-f92e-45bd-84fd-a54422ce803b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:21.180343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-c7cb7a6e-f92e-45bd-84fd-a54422ce803b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:21.180343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:21.180343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:21.180343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:21.180343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:21.180343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:21.180343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:21.180343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:21.180343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:21.180343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:21.180343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:21.180343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:21.180343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:21.180343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:21.180343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:21.180343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:21.182209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:21.182209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:21.182209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:21.182209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:21.182209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:21.182209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:21.182209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:21.182209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:21.182209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:21.182209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:21.182209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:21.182209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:21.182209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:21.182209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:21.182209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:21.182209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:21.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:21.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:21.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:21.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:21.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:21.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:21.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:21.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:21.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:21.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:21.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:21.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:21.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 283/562] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:43:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:21.183756 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cb7a6e-f92e-45bd-84fd-a54422ce803b None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:21.288596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:21.290718 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cb7a6e-f92e-45bd-84fd-a54422ce803b None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:21.310891 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b7e6a9-450b-452b-8b6d-48ff9301ee1d None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:21.339001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f322442-a2dd-4657-a70c-c99a95264fea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:21.341719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f322442-a2dd-4657-a70c-c99a95264fea None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:21.342052 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f322442-a2dd-4657-a70c-c99a95264fea None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:21.342256 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f322442-a2dd-4657-a70c-c99a95264fea None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:21.359655 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d176eb-22db-45a7-8719-dd8347481b2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:21.360303 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d176eb-22db-45a7-8719-dd8347481b2f None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:21.360584 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d176eb-22db-45a7-8719-dd8347481b2f None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:21.360848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d176eb-22db-45a7-8719-dd8347481b2f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:21.363567 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:21.374802 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d176eb-22db-45a7-8719-dd8347481b2f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:21.384147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-e0d176eb-22db-45a7-8719-dd8347481b2f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:21.384147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:21.384147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:21.384147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:21.384147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:21.384147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:21.384147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:21.384147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:21.384147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:21.384147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:21.384147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:21.384147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:21.384147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:21.384147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:21.384147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:21.384147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:21.385168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:21.385168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:21.385168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:21.385168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:21.385168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:21.385168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:21.385168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:21.385168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:21.385168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:21.385168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:21.385168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:21.385168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:21.385168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:21.385168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:21.385168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:21.385168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:21.386001 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:21.386001 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:21.386001 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:21.386001 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:21.386001 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:21.386001 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:21.386001 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:21.386001 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:21.386001 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:21.386001 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:21.386001 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:21.386001 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:21.386001 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 280/563] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:43:21 2022] GET /identity/v3/roles/service => generated 85 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:21.387332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d176eb-22db-45a7-8719-dd8347481b2f None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:21.444492 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f322442-a2dd-4657-a70c-c99a95264fea None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:21.446056 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f322442-a2dd-4657-a70c-c99a95264fea None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:21.546523 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d176eb-22db-45a7-8719-dd8347481b2f None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:21.580620 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 284/564] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:43:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:43:21.582120 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f322442-a2dd-4657-a70c-c99a95264fea None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:21.587462 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1abc2c2-2977-46e0-840b-7c7b159d744d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:21.588106 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1abc2c2-2977-46e0-840b-7c7b159d744d None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:21.588381 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1abc2c2-2977-46e0-840b-7c7b159d744d None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:21.588651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1abc2c2-2977-46e0-840b-7c7b159d744d None admin] PATH_INFO: `/v3/users/glance` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:21.597949 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1abc2c2-2977-46e0-840b-7c7b159d744d None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:21.599456 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1abc2c2-2977-46e0-840b-7c7b159d744d None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:21.601667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:21.613729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-d1abc2c2-2977-46e0-840b-7c7b159d744d None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 18 00:43:21.613729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:21.613729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:21.613729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:21.613729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:21.613729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:21.613729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:21.613729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:21.613729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:21.613729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:21.613729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:21.613729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:21.613729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:43:21.613729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:43:21.613729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:43:21.613729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:43:21.615521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:21.615521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:21.615521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:43:21.615521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:21.615521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:43:21.615521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:21.615521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:21.615521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:21.615521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:21.615521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:21.615521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:21.615521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:21.615521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:21.615521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:21.615521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:21.615521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:21.615521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:21.615521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:21.617212 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:21.617212 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:21.617212 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:43:21.617212 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:43:21.617212 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:43:21.617212 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:43:21.617212 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:43:21.617212 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:43:21.617212 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 18 00:43:21.617212 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:21.617212 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 281/565] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:43:21 2022] GET /identity/v3/users/glance => generated 84 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:21.617212 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1abc2c2-2977-46e0-840b-7c7b159d744d None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:21.649974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:21.730390 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f322442-a2dd-4657-a70c-c99a95264fea None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:21.750046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1abc2c2-2977-46e0-840b-7c7b159d744d None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:21.769200 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5e80fe-94b7-4fa1-88cb-010436efc166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:21.769948 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5e80fe-94b7-4fa1-88cb-010436efc166 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:21.770081 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5e80fe-94b7-4fa1-88cb-010436efc166 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:21.770320 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5e80fe-94b7-4fa1-88cb-010436efc166 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:21.771025 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5e80fe-94b7-4fa1-88cb-010436efc166 None admin] RBAC: Adding query filter params (name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:21.771321 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5e80fe-94b7-4fa1-88cb-010436efc166 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:21.772683 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5e80fe-94b7-4fa1-88cb-010436efc166 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:21.786414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 285/566] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 18 00:43:21 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:21.787366 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5e80fe-94b7-4fa1-88cb-010436efc166 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:21.794408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237d7c19-8a2b-4792-a11c-afddc66b47c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:21.795140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237d7c19-8a2b-4792-a11c-afddc66b47c0 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:21.795457 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237d7c19-8a2b-4792-a11c-afddc66b47c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:21.795769 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237d7c19-8a2b-4792-a11c-afddc66b47c0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:21.812085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237d7c19-8a2b-4792-a11c-afddc66b47c0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:21.822818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-237d7c19-8a2b-4792-a11c-afddc66b47c0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:21.822818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:21.822818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:21.822818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:21.822818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:21.822818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:21.822818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:21.822818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:21.822818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:21.822818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:21.822818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:21.822818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:21.822818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:21.822818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:21.822818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:21.822818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:21.824909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:21.824909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:21.824909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:21.824909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:21.824909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:21.824909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:21.824909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:21.824909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:21.824909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:21.824909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:21.824909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:21.824909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:21.824909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:21.824909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:21.824909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:21.824909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:21.825741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:21.825741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:21.825741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:21.825741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:21.825741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:21.825741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:21.825741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:21.825741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:21.825741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:21.825741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:21.825741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:21.825741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:21.825741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 282/567] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:43:21 2022] GET /identity/v3/roles/service => generated 85 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:21.826960 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237d7c19-8a2b-4792-a11c-afddc66b47c0 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:21.901883 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5e80fe-94b7-4fa1-88cb-010436efc166 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:21.916143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:21.923538 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:21.945228 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b92f77d-22d5-4194-ba0b-8622ceddd65a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:21.945872 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b92f77d-22d5-4194-ba0b-8622ceddd65a None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:21.946277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b92f77d-22d5-4194-ba0b-8622ceddd65a None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:21.946416 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b92f77d-22d5-4194-ba0b-8622ceddd65a None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:21.947274 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b92f77d-22d5-4194-ba0b-8622ceddd65a None admin] RBAC: Adding query filter params (name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:21.947574 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b92f77d-22d5-4194-ba0b-8622ceddd65a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:21.949627 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b92f77d-22d5-4194-ba0b-8622ceddd65a None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:21.959565 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 286/568] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 18 00:43:21 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:21.960470 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b92f77d-22d5-4194-ba0b-8622ceddd65a None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:22.003873 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237d7c19-8a2b-4792-a11c-afddc66b47c0 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:22.052168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6039f86f-8208-4196-9849-d422daaece3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:22.052887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6039f86f-8208-4196-9849-d422daaece3b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:22.053234 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6039f86f-8208-4196-9849-d422daaece3b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:22.053574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6039f86f-8208-4196-9849-d422daaece3b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:22.066933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6039f86f-8208-4196-9849-d422daaece3b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:22.068447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6039f86f-8208-4196-9849-d422daaece3b None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:22.086056 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b92f77d-22d5-4194-ba0b-8622ceddd65a None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:22.087486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-6039f86f-8208-4196-9849-d422daaece3b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:22.087486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:22.087486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:22.087486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:22.087486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:22.087486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:22.087486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:22.087486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:22.087486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:22.087486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:22.087486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:22.087486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:22.087486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:22.087486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:22.087486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:22.087486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:22.089433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:22.089433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:22.089433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:22.089433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:22.089433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:22.089433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:22.089433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:22.089433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:22.089433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:22.089433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:22.089433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:22.089433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:22.089433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:22.089433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:22.089433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:22.089433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:22.089433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:22.091800 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:22.091800 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:22.091800 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:22.091800 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 283/569] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:22.091800 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6039f86f-8208-4196-9849-d422daaece3b None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:22.141440 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbfac45-2cc4-405e-ad3f-b90417c9ce64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:22.142201 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbfac45-2cc4-405e-ad3f-b90417c9ce64 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:22.142650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbfac45-2cc4-405e-ad3f-b90417c9ce64 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:22.143024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbfac45-2cc4-405e-ad3f-b90417c9ce64 None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:22.143855 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbfac45-2cc4-405e-ad3f-b90417c9ce64 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:22.144248 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbfac45-2cc4-405e-ad3f-b90417c9ce64 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:22.145984 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbfac45-2cc4-405e-ad3f-b90417c9ce64 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:22.166649 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 287/570] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Sep 18 00:43:21 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:22.167462 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbfac45-2cc4-405e-ad3f-b90417c9ce64 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:22.199707 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:22.221506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6039f86f-8208-4196-9849-d422daaece3b None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:22.238626 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:22.263778 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d995f6-40d1-4e7d-a935-024e245f35fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:22.264420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d995f6-40d1-4e7d-a935-024e245f35fa None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:22.264698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d995f6-40d1-4e7d-a935-024e245f35fa None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:22.264970 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d995f6-40d1-4e7d-a935-024e245f35fa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:22.277655 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d995f6-40d1-4e7d-a935-024e245f35fa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:22.279201 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d995f6-40d1-4e7d-a935-024e245f35fa None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:22.295069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-b9d995f6-40d1-4e7d-a935-024e245f35fa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:22.295069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:22.295069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:22.295069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:22.295069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:22.295069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:22.295069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:22.295069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:22.295069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:22.295069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:22.295069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:22.295069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:22.295069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:22.295069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:22.295069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:22.295069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:22.296978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:22.296978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:22.296978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:22.296978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:22.296978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:22.296978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:22.296978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:22.296978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:22.296978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:22.296978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:22.296978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:22.296978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:22.296978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:22.296978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:22.296978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:22.296978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:22.296978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:22.298804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:22.298804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:22.298804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:22.298804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 284/571] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:22.302019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d995f6-40d1-4e7d-a935-024e245f35fa None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:22.311220 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbfac45-2cc4-405e-ad3f-b90417c9ce64 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:22.371621 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d471aa-35e1-43cf-b596-874db3327ab8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:22.372622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d471aa-35e1-43cf-b596-874db3327ab8 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:22.373115 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d471aa-35e1-43cf-b596-874db3327ab8 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:22.373596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d471aa-35e1-43cf-b596-874db3327ab8 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:22.374829 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d471aa-35e1-43cf-b596-874db3327ab8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:22.375383 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d471aa-35e1-43cf-b596-874db3327ab8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:22.378001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d471aa-35e1-43cf-b596-874db3327ab8 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:22.389362 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 288/572] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 18 00:43:22 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:22.390808 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d471aa-35e1-43cf-b596-874db3327ab8 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:22.423681 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d995f6-40d1-4e7d-a935-024e245f35fa None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:22.464643 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11720297-1798-43bc-a792-02b350aae161 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:22.465423 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11720297-1798-43bc-a792-02b350aae161 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:22.465791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11720297-1798-43bc-a792-02b350aae161 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:22.466156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11720297-1798-43bc-a792-02b350aae161 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:22.479719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11720297-1798-43bc-a792-02b350aae161 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:22.481866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11720297-1798-43bc-a792-02b350aae161 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:22.500228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-11720297-1798-43bc-a792-02b350aae161 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:22.500228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:22.500228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:22.500228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:22.500228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:22.500228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:22.500228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:22.500228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:22.500228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:22.500228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:22.500228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:22.500228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:22.500228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:22.500228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:22.500228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:22.500228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:22.501958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:22.501958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:22.501958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:22.501958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:22.501958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:22.501958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:22.501958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:22.501958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:22.501958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:22.501958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:22.501958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:22.501958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:22.501958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:22.501958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:22.501958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:22.501958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:22.501958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:22.504137 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:22.504137 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:22.504137 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:22.504137 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 285/573] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:22.504137 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11720297-1798-43bc-a792-02b350aae161 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:22.506488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:22.520027 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d471aa-35e1-43cf-b596-874db3327ab8 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:22.529532 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:22.580349 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148e9418-e124-42a7-8fd0-a5862729ce76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:22.581421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148e9418-e124-42a7-8fd0-a5862729ce76 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:22.581897 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148e9418-e124-42a7-8fd0-a5862729ce76 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:22.582385 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148e9418-e124-42a7-8fd0-a5862729ce76 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:22.583593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148e9418-e124-42a7-8fd0-a5862729ce76 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:22.584136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148e9418-e124-42a7-8fd0-a5862729ce76 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:22.586831 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148e9418-e124-42a7-8fd0-a5862729ce76 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:22.607865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 289/574] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:22.608782 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148e9418-e124-42a7-8fd0-a5862729ce76 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:22.640209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11720297-1798-43bc-a792-02b350aae161 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:22.694319 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871dc912-fe50-49f2-b207-18ba177d683f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:22.695261 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871dc912-fe50-49f2-b207-18ba177d683f None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:22.695707 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871dc912-fe50-49f2-b207-18ba177d683f None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:22.696149 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871dc912-fe50-49f2-b207-18ba177d683f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:22.715155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871dc912-fe50-49f2-b207-18ba177d683f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:22.717711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871dc912-fe50-49f2-b207-18ba177d683f None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:22.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-871dc912-fe50-49f2-b207-18ba177d683f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:22.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:22.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:22.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:22.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:22.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:22.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:22.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:22.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:22.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:22.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:22.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:22.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:22.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:22.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:22.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:22.738417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:22.738417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:22.738417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:22.738417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:22.738417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:22.738417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:22.738417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:22.738417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:22.738417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:22.738417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:22.738417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:22.738417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:22.738417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:22.738417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:22.738417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:22.738417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:22.738417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:22.740796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:22.740796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:22.740796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:22.740796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 286/575] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:22.740796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871dc912-fe50-49f2-b207-18ba177d683f None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:22.763575 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148e9418-e124-42a7-8fd0-a5862729ce76 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:22.812539 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b990f6dd-6468-49fa-9d16-4278ca285336 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:22.813512 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b990f6dd-6468-49fa-9d16-4278ca285336 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:22.813868 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b990f6dd-6468-49fa-9d16-4278ca285336 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:22.814314 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b990f6dd-6468-49fa-9d16-4278ca285336 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:22.815466 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b990f6dd-6468-49fa-9d16-4278ca285336 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:22.815965 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b990f6dd-6468-49fa-9d16-4278ca285336 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:22.818386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b990f6dd-6468-49fa-9d16-4278ca285336 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:22.828791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:22.836680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 290/576] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:22.837907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b990f6dd-6468-49fa-9d16-4278ca285336 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:22.855413 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:22.874760 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871dc912-fe50-49f2-b207-18ba177d683f None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:22.913794 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce89070a-9c48-4fc0-b654-64d105e4a4b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:22.914439 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce89070a-9c48-4fc0-b654-64d105e4a4b6 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:22.914761 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce89070a-9c48-4fc0-b654-64d105e4a4b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:22.915038 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce89070a-9c48-4fc0-b654-64d105e4a4b6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:22.926950 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce89070a-9c48-4fc0-b654-64d105e4a4b6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:22.928460 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce89070a-9c48-4fc0-b654-64d105e4a4b6 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:22.944076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-ce89070a-9c48-4fc0-b654-64d105e4a4b6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:22.944076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:22.944076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:22.944076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:22.944076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:22.944076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:22.944076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:22.944076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:22.944076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:22.944076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:22.944076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:22.944076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:22.944076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:22.944076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:22.944076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:22.944076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:22.945860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:22.945860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:22.945860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:22.945860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:22.945860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:22.945860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:22.945860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:22.945860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:22.945860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:22.945860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:22.945860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:22.945860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:22.945860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:22.945860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:22.945860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:22.945860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:22.945860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:22.946902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:22.946902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:22.946902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:22.946902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 287/577] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:22.946902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce89070a-9c48-4fc0-b654-64d105e4a4b6 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:22.979177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b990f6dd-6468-49fa-9d16-4278ca285336 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:23.017148 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafcda7a-d08a-4223-9d5a-fac91c1c984b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:23.017776 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafcda7a-d08a-4223-9d5a-fac91c1c984b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:23.018052 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafcda7a-d08a-4223-9d5a-fac91c1c984b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:23.018320 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafcda7a-d08a-4223-9d5a-fac91c1c984b None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:23.018996 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafcda7a-d08a-4223-9d5a-fac91c1c984b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:23.019298 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafcda7a-d08a-4223-9d5a-fac91c1c984b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:23.020673 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafcda7a-d08a-4223-9d5a-fac91c1c984b None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:23.033679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 291/578] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:23.034550 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafcda7a-d08a-4223-9d5a-fac91c1c984b None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:23.099478 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce89070a-9c48-4fc0-b654-64d105e4a4b6 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:23.134958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:23.151066 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:23.161058 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82be217-fed5-4eee-900a-428272957acc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:23.162036 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82be217-fed5-4eee-900a-428272957acc None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:23.162509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82be217-fed5-4eee-900a-428272957acc None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:23.162865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82be217-fed5-4eee-900a-428272957acc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:23.174209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82be217-fed5-4eee-900a-428272957acc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:23.176137 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82be217-fed5-4eee-900a-428272957acc None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:23.189344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-b82be217-fed5-4eee-900a-428272957acc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:23.189344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:23.189344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:23.189344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:23.189344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:23.189344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:23.189344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:23.189344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:23.189344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:23.189344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:23.189344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:23.189344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:23.189344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:43:23.189344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:43:23.189344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:43:23.191125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:43:23.191125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:23.191125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:23.191125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:23.191125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:23.191125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:23.191125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:23.191125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:23.191125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:23.191125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:23.191125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:23.191125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:23.191125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:23.191125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:23.191125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:23.191125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:23.191125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:23.193127 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:43:23.193127 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:43:23.193127 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:43:23.193127 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:43:23.193127 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:43:23.193127 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:43:23.193127 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:23.193127 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:23.193127 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 288/579] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:23.193127 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82be217-fed5-4eee-900a-428272957acc None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:23.201422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafcda7a-d08a-4223-9d5a-fac91c1c984b None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:23.244582 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dcb5ba-02d0-49e3-95a7-50bddb188e61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:23.245610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dcb5ba-02d0-49e3-95a7-50bddb188e61 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:23.245994 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dcb5ba-02d0-49e3-95a7-50bddb188e61 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:23.246433 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dcb5ba-02d0-49e3-95a7-50bddb188e61 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:23.247234 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8dcb5ba-02d0-49e3-95a7-50bddb188e61 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:23.247646 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8dcb5ba-02d0-49e3-95a7-50bddb188e61 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:23.249269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8dcb5ba-02d0-49e3-95a7-50bddb188e61 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:23.263011 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 292/580] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:23.264497 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dcb5ba-02d0-49e3-95a7-50bddb188e61 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:23.321311 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82be217-fed5-4eee-900a-428272957acc None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:23.364466 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a296c4-a380-42f7-a17d-5189e8dfec94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:23.365268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a296c4-a380-42f7-a17d-5189e8dfec94 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:23.365629 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a296c4-a380-42f7-a17d-5189e8dfec94 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:23.365985 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a296c4-a380-42f7-a17d-5189e8dfec94 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:23.374935 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a296c4-a380-42f7-a17d-5189e8dfec94 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:23.376408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a296c4-a380-42f7-a17d-5189e8dfec94 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:23.389346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-28a296c4-a380-42f7-a17d-5189e8dfec94 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 18 00:43:23.389346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:23.389346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:23.389346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:23.389346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:23.389346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:23.389346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:23.389346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:23.389346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:23.389346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:23.389346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:23.389346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:23.389346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:43:23.389346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:43:23.389346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:43:23.389346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:43:23.391290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:23.391290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:23.391290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:43:23.391290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:23.391290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:43:23.391290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:23.391290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:23.391290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:23.391290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:23.391290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:23.391290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:23.391290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:23.391290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:23.391290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:23.391290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:23.391290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:23.391290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:23.391290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:23.393216 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:23.393216 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:23.393216 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:43:23.393216 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:43:23.393216 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:43:23.393216 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:43:23.393216 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:43:23.393216 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:43:23.393216 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 18 00:43:23.393216 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:23.393216 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 289/581] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Sep 18 00:43:23 2022] GET /identity/v3/users/nova => generated 82 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:23.393216 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a296c4-a380-42f7-a17d-5189e8dfec94 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:23.403138 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dcb5ba-02d0-49e3-95a7-50bddb188e61 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:23.437200 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:23.443033 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:23.447193 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22b8039-598b-4667-8847-24af7c61b6d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:23.448120 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22b8039-598b-4667-8847-24af7c61b6d0 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:23.449077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22b8039-598b-4667-8847-24af7c61b6d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:23.449532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22b8039-598b-4667-8847-24af7c61b6d0 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:23.450650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22b8039-598b-4667-8847-24af7c61b6d0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:23.451115 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22b8039-598b-4667-8847-24af7c61b6d0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:23.453677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22b8039-598b-4667-8847-24af7c61b6d0 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:23.473915 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 293/582] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:23.475537 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22b8039-598b-4667-8847-24af7c61b6d0 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:23.522013 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a296c4-a380-42f7-a17d-5189e8dfec94 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:23.562353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d5dd06-f927-4fdf-81e3-31262f3d50ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:23.563357 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d5dd06-f927-4fdf-81e3-31262f3d50ea None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:23.563810 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d5dd06-f927-4fdf-81e3-31262f3d50ea None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:23.564264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d5dd06-f927-4fdf-81e3-31262f3d50ea None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:23.577468 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d5dd06-f927-4fdf-81e3-31262f3d50ea None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:23.580010 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d5dd06-f927-4fdf-81e3-31262f3d50ea None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:23.599866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-01d5dd06-f927-4fdf-81e3-31262f3d50ea None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 18 00:43:23.599866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:23.599866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:23.599866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:23.599866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:23.599866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:23.599866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:23.599866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:23.599866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:23.599866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:23.599866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:23.599866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:23.599866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:43:23.599866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:43:23.599866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:43:23.599866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:43:23.601824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:23.601824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:23.601824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:43:23.601824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:23.601824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:43:23.601824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:23.601824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:23.601824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:23.601824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:23.601824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:23.601824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:23.601824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:23.601824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:23.601824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:23.601824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:23.601824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:23.601824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:23.601824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:23.603428 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:23.603428 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:23.603428 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:43:23.603428 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:43:23.603428 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:43:23.603428 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:43:23.603428 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:43:23.603428 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:43:23.603428 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 18 00:43:23.603428 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:23.603428 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 290/583] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:43:23 2022] GET /identity/v3/users/neutron => generated 85 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:23.604646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d5dd06-f927-4fdf-81e3-31262f3d50ea None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:23.639286 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22b8039-598b-4667-8847-24af7c61b6d0 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:23.685381 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c042ec7f-98ce-4af2-bcab-19b5b3a90f6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:23.686127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c042ec7f-98ce-4af2-bcab-19b5b3a90f6f None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:23.686557 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c042ec7f-98ce-4af2-bcab-19b5b3a90f6f None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:23.686985 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c042ec7f-98ce-4af2-bcab-19b5b3a90f6f None admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:23.687919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c042ec7f-98ce-4af2-bcab-19b5b3a90f6f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:23.688342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c042ec7f-98ce-4af2-bcab-19b5b3a90f6f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:23.689926 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c042ec7f-98ce-4af2-bcab-19b5b3a90f6f None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:23.705771 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 294/584] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:43:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:23.706797 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c042ec7f-98ce-4af2-bcab-19b5b3a90f6f None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:23.737104 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:23.743954 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d5dd06-f927-4fdf-81e3-31262f3d50ea None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:23.773291 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:23.794733 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e328b0ba-d0e4-406f-a0d1-d2d93f06cc3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:23.795868 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e328b0ba-d0e4-406f-a0d1-d2d93f06cc3a None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:23.796118 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e328b0ba-d0e4-406f-a0d1-d2d93f06cc3a None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:23.796400 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e328b0ba-d0e4-406f-a0d1-d2d93f06cc3a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:23.804399 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e328b0ba-d0e4-406f-a0d1-d2d93f06cc3a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:23.805900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e328b0ba-d0e4-406f-a0d1-d2d93f06cc3a None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:23.818088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-e328b0ba-d0e4-406f-a0d1-d2d93f06cc3a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:23.818088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:23.818088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:23.818088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:23.818088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:23.818088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:23.818088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:23.818088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:23.818088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:23.818088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:23.818088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:23.818088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:23.818088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:43:23.818088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:43:23.818088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:43:23.820034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:43:23.820034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:23.820034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:23.820034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:23.820034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:23.820034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:23.820034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:23.820034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:23.820034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:23.820034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:23.820034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:23.820034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:23.820034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:23.820034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:23.820034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:23.820034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:23.820034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:23.821937 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:43:23.821937 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:43:23.821937 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:43:23.821937 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:43:23.821937 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:43:23.821937 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:43:23.821937 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:23.821937 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:23.821937 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 291/585] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:23.821937 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e328b0ba-d0e4-406f-a0d1-d2d93f06cc3a None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:23.856793 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c042ec7f-98ce-4af2-bcab-19b5b3a90f6f None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:23.900005 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd3d6c4-462f-44e2-a03a-5e204d629bed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:23.900791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd3d6c4-462f-44e2-a03a-5e204d629bed None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:23.901160 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd3d6c4-462f-44e2-a03a-5e204d629bed None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:23.901545 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd3d6c4-462f-44e2-a03a-5e204d629bed None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:23.902968 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd3d6c4-462f-44e2-a03a-5e204d629bed None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:23.903381 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd3d6c4-462f-44e2-a03a-5e204d629bed None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:23.904920 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd3d6c4-462f-44e2-a03a-5e204d629bed None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:23.929784 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 295/586] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Sep 18 00:43:23 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:23.930894 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd3d6c4-462f-44e2-a03a-5e204d629bed None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:23.960020 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e328b0ba-d0e4-406f-a0d1-d2d93f06cc3a None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:24.011021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a56144-2f8f-42d9-9adf-5a5415e3d8fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:24.012307 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a56144-2f8f-42d9-9adf-5a5415e3d8fe None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:24.012661 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a56144-2f8f-42d9-9adf-5a5415e3d8fe None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:24.013013 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a56144-2f8f-42d9-9adf-5a5415e3d8fe None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:24.023210 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a56144-2f8f-42d9-9adf-5a5415e3d8fe None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:24.024865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a56144-2f8f-42d9-9adf-5a5415e3d8fe None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:24.038274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-52a56144-2f8f-42d9-9adf-5a5415e3d8fe None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 18 00:43:24.038274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:24.038274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:24.038274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:24.038274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:24.038274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:24.038274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:24.038274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:24.038274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:24.038274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:24.038274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:24.038274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:24.038274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:43:24.038274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:43:24.038274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:43:24.038274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:43:24.039811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:24.039811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:24.039811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:43:24.039811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:24.039811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:43:24.039811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:24.039811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:24.039811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:24.039811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:24.039811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:24.039811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:24.039811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:24.039811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:24.039811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:24.039811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:24.039811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:24.039811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:24.039811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:24.041908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:24.041908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:24.041908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:43:24.041908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:43:24.041908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:43:24.041908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:43:24.041908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:43:24.041908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:43:24.041908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 18 00:43:24.041908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:24.041908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 292/587] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:43:23 2022] GET /identity/v3/users/cinder => generated 84 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:24.041908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a56144-2f8f-42d9-9adf-5a5415e3d8fe None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:24.069956 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:24.075637 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd3d6c4-462f-44e2-a03a-5e204d629bed None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:24.088052 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:24.128892 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9615823-015f-4488-880f-50c57f6b859d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:24.129618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9615823-015f-4488-880f-50c57f6b859d None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:24.130050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9615823-015f-4488-880f-50c57f6b859d None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:24.130539 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9615823-015f-4488-880f-50c57f6b859d None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:24.131520 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9615823-015f-4488-880f-50c57f6b859d None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:24.132033 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9615823-015f-4488-880f-50c57f6b859d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:24.133757 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9615823-015f-4488-880f-50c57f6b859d None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:24.156725 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 296/588] 10.222.0.42 () {56 vars in 1139 bytes} [Sun Sep 18 00:43:23 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:24.158123 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9615823-015f-4488-880f-50c57f6b859d None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:24.183982 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a56144-2f8f-42d9-9adf-5a5415e3d8fe None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:24.232977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d6b821-172d-40c0-bf27-726c5c14bd2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:24.233507 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d6b821-172d-40c0-bf27-726c5c14bd2c None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:24.233782 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d6b821-172d-40c0-bf27-726c5c14bd2c None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:24.234050 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d6b821-172d-40c0-bf27-726c5c14bd2c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:24.234815 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d6b821-172d-40c0-bf27-726c5c14bd2c None admin] RBAC: Adding query filter params (role.id=81523e544c7e4993af4d1e06ce3754bc, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=6230ed7ce0eb43e9b2b8530a28eb6299) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:24.235126 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d6b821-172d-40c0-bf27-726c5c14bd2c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:24.236596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d6b821-172d-40c0-bf27-726c5c14bd2c None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:24.242628 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 293/589] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:43:24 2022] GET /identity/v3/role_assignments?user.id=6230ed7ce0eb43e9b2b8530a28eb6299&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=81523e544c7e4993af4d1e06ce3754bc => generated 259 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:24.243833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d6b821-172d-40c0-bf27-726c5c14bd2c None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:24.327919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9615823-015f-4488-880f-50c57f6b859d None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:24.352255 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:24.376642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d6b821-172d-40c0-bf27-726c5c14bd2c None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:24.403401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69cc61d8-208a-4428-8391-741eaae1b666 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:24.404522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cc61d8-208a-4428-8391-741eaae1b666 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:24.405015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cc61d8-208a-4428-8391-741eaae1b666 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:24.405524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cc61d8-208a-4428-8391-741eaae1b666 None admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:24.406916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69cc61d8-208a-4428-8391-741eaae1b666 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:24.407473 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69cc61d8-208a-4428-8391-741eaae1b666 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:24.411501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69cc61d8-208a-4428-8391-741eaae1b666 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:24.429941 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec436e8d-2b5a-4c51-b0ba-d901ddba9dfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:24.430609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec436e8d-2b5a-4c51-b0ba-d901ddba9dfa None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:24.430886 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec436e8d-2b5a-4c51-b0ba-d901ddba9dfa None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:24.431156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec436e8d-2b5a-4c51-b0ba-d901ddba9dfa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:24.444442 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 297/590] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:43:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:24.445185 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69cc61d8-208a-4428-8391-741eaae1b666 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:24.446397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec436e8d-2b5a-4c51-b0ba-d901ddba9dfa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:24.447886 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec436e8d-2b5a-4c51-b0ba-d901ddba9dfa None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:24.469657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-ec436e8d-2b5a-4c51-b0ba-d901ddba9dfa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:24.469657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:24.469657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:24.469657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:24.469657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:24.469657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:24.469657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:24.469657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:24.469657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:24.469657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:24.469657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:24.469657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:24.469657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:24.469657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:24.469657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:24.469657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:24.481675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:24.481675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:24.481675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:24.481675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:24.481675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:24.481675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:24.481675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:24.481675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:24.481675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:24.481675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:24.481675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:24.481675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:24.481675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:24.481675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:24.481675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:24.481675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:24.481675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:24.483713 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:24.483713 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:24.483713 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:24.483713 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 294/591] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:24.483713 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec436e8d-2b5a-4c51-b0ba-d901ddba9dfa None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:24.513402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:24.641122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69cc61d8-208a-4428-8391-741eaae1b666 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:24.670235 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec436e8d-2b5a-4c51-b0ba-d901ddba9dfa None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:24.684192 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b747d99f-4313-4aa1-8ade-28116951db74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:24.685255 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b747d99f-4313-4aa1-8ade-28116951db74 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:24.685571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b747d99f-4313-4aa1-8ade-28116951db74 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:24.685970 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b747d99f-4313-4aa1-8ade-28116951db74 None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:24.686925 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b747d99f-4313-4aa1-8ade-28116951db74 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:24.687346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b747d99f-4313-4aa1-8ade-28116951db74 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:24.688919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b747d99f-4313-4aa1-8ade-28116951db74 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:24.717580 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 298/592] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Sep 18 00:43:24 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:24.717580 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b747d99f-4313-4aa1-8ade-28116951db74 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:24.746660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a6155e-ddd0-452c-99f7-e1a6de0d8e41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:24.747781 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a6155e-ddd0-452c-99f7-e1a6de0d8e41 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:24.748400 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a6155e-ddd0-452c-99f7-e1a6de0d8e41 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:24.748945 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a6155e-ddd0-452c-99f7-e1a6de0d8e41 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:24.754437 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:24.773696 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a6155e-ddd0-452c-99f7-e1a6de0d8e41 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:24.776672 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a6155e-ddd0-452c-99f7-e1a6de0d8e41 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:24.804436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-54a6155e-ddd0-452c-99f7-e1a6de0d8e41 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:24.804436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:24.804436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:24.804436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:24.804436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:24.804436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:24.804436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:24.804436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:24.804436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:24.804436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:24.804436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:24.804436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:24.804436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:24.804436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:24.804436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:24.804436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:24.805793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:24.805793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:24.805793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:24.805793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:24.805793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:24.805793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:24.805793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:24.805793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:24.805793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:24.805793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:24.805793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:24.805793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:24.805793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:24.805793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:24.805793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:24.805793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:24.805793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:24.807233 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:24.807233 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:24.807233 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:24.815045 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 295/593] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 335 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:24.815045 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a6155e-ddd0-452c-99f7-e1a6de0d8e41 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:24.828311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:24.848898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b747d99f-4313-4aa1-8ade-28116951db74 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:24.895063 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbbfbbd-1e7c-41ec-9eae-387f47604fdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:24.896486 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbbfbbd-1e7c-41ec-9eae-387f47604fdf None admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:24.896780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbbfbbd-1e7c-41ec-9eae-387f47604fdf None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:24.897037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbbfbbd-1e7c-41ec-9eae-387f47604fdf None admin] PATH_INFO: `/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/d7fe9d21099e4c4891c28687bb673214/roles/81523e544c7e4993af4d1e06ce3754bc` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:24.917069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbbfbbd-1e7c-41ec-9eae-387f47604fdf None admin] RBAC: Authorizing `identity:create_grant(project_id=c520a8da3dc54adba4f7cc04ef42e367, user_id=d7fe9d21099e4c4891c28687bb673214, role_id=81523e544c7e4993af4d1e06ce3754bc)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:24.920377 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbbfbbd-1e7c-41ec-9eae-387f47604fdf None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:24.941423 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 299/594] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:43:24 2022] PUT /identity/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/d7fe9d21099e4c4891c28687bb673214/roles/81523e544c7e4993af4d1e06ce3754bc => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:43:24.942190 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbbfbbd-1e7c-41ec-9eae-387f47604fdf None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:24.969057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a6155e-ddd0-452c-99f7-e1a6de0d8e41 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:25.032720 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0213ab-428b-4cf5-a4cb-dbacbab2adf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:25.034176 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0213ab-428b-4cf5-a4cb-dbacbab2adf7 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:25.035027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0213ab-428b-4cf5-a4cb-dbacbab2adf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:25.035841 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0213ab-428b-4cf5-a4cb-dbacbab2adf7 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:25.037842 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0213ab-428b-4cf5-a4cb-dbacbab2adf7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:25.038832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0213ab-428b-4cf5-a4cb-dbacbab2adf7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:25.041576 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0213ab-428b-4cf5-a4cb-dbacbab2adf7 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:25.071326 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 296/595] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:25.073526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0213ab-428b-4cf5-a4cb-dbacbab2adf7 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:25.081989 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbbfbbd-1e7c-41ec-9eae-387f47604fdf None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:25.142879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351ecab7-4108-48cc-85d2-79d7828e2dfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:25.144368 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351ecab7-4108-48cc-85d2-79d7828e2dfd None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:25.145182 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351ecab7-4108-48cc-85d2-79d7828e2dfd None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:25.145967 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351ecab7-4108-48cc-85d2-79d7828e2dfd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:25.149311 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:25.149934 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:25.163651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351ecab7-4108-48cc-85d2-79d7828e2dfd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:25.165149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351ecab7-4108-48cc-85d2-79d7828e2dfd None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:25.182663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-351ecab7-4108-48cc-85d2-79d7828e2dfd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:25.182663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:25.182663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:25.182663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:25.182663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:25.182663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:25.182663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:25.182663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:25.182663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:25.182663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:25.182663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:25.182663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:25.182663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:25.182663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:25.182663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:25.182663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:25.185073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:25.185073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:25.185073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:25.185073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:25.185073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:25.185073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:25.185073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:25.185073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:25.185073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:25.185073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:25.185073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:25.185073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:25.185073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:25.185073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:25.185073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:25.185073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:25.185073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:25.186843 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:25.186843 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:25.186843 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:25.186843 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 300/596] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:25.186843 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351ecab7-4108-48cc-85d2-79d7828e2dfd None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:25.232917 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0213ab-428b-4cf5-a4cb-dbacbab2adf7 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:25.270557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30885404-3eca-471a-9e60-306729f86fc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:25.271185 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30885404-3eca-471a-9e60-306729f86fc4 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:25.271469 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30885404-3eca-471a-9e60-306729f86fc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:25.271726 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30885404-3eca-471a-9e60-306729f86fc4 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:25.272395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30885404-3eca-471a-9e60-306729f86fc4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:25.272676 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30885404-3eca-471a-9e60-306729f86fc4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:25.274099 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30885404-3eca-471a-9e60-306729f86fc4 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:25.287917 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 297/597] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:25.288705 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30885404-3eca-471a-9e60-306729f86fc4 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:25.307457 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351ecab7-4108-48cc-85d2-79d7828e2dfd None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:25.346908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ecd961-808e-4903-9dbd-d77daa84395a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:25.347575 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ecd961-808e-4903-9dbd-d77daa84395a None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:25.347922 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ecd961-808e-4903-9dbd-d77daa84395a None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:25.348216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ecd961-808e-4903-9dbd-d77daa84395a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:25.356051 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ecd961-808e-4903-9dbd-d77daa84395a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:25.357642 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ecd961-808e-4903-9dbd-d77daa84395a None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:25.371105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-b8ecd961-808e-4903-9dbd-d77daa84395a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:25.371105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:25.371105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:25.371105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:25.371105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:25.371105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:25.371105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:25.371105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:25.371105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:25.371105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:25.371105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:25.371105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:25.371105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:43:25.371105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:43:25.371105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:43:25.374615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:43:25.374615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:25.374615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:25.374615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:25.374615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:25.374615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:25.374615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:25.374615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:25.374615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:25.374615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:25.374615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:25.374615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:25.374615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:25.374615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:25.374615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:25.374615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:25.374615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:25.377821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:43:25.377821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:43:25.377821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:43:25.377821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:43:25.377821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:43:25.377821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:43:25.377821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:25.377821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:25.377821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 301/598] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:25.377821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ecd961-808e-4903-9dbd-d77daa84395a None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:25.424881 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30885404-3eca-471a-9e60-306729f86fc4 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:25.436410 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:25.476049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f950486-dddd-4772-b683-a4f8c8389da6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:25.476701 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f950486-dddd-4772-b683-a4f8c8389da6 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:25.476983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f950486-dddd-4772-b683-a4f8c8389da6 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:25.477258 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f950486-dddd-4772-b683-a4f8c8389da6 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:25.477941 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f950486-dddd-4772-b683-a4f8c8389da6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:25.478239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f950486-dddd-4772-b683-a4f8c8389da6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:25.481406 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f950486-dddd-4772-b683-a4f8c8389da6 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:25.509215 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 298/599] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:25.511923 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:25.512937 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f950486-dddd-4772-b683-a4f8c8389da6 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:25.604879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ecd961-808e-4903-9dbd-d77daa84395a None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:25.654524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f950486-dddd-4772-b683-a4f8c8389da6 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:25.661051 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd360346-b77c-435e-ab76-179498b0f2bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:25.662118 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd360346-b77c-435e-ab76-179498b0f2bf None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:25.662646 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd360346-b77c-435e-ab76-179498b0f2bf None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:25.663194 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd360346-b77c-435e-ab76-179498b0f2bf None admin] PATH_INFO: `/v3/projects/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:25.671257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd360346-b77c-435e-ab76-179498b0f2bf None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:25.673096 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd360346-b77c-435e-ab76-179498b0f2bf None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:25.686168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-bd360346-b77c-435e-ab76-179498b0f2bf None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:25.686168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:25.686168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:25.686168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:25.686168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:25.686168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:25.686168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:25.686168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:25.686168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:25.686168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:25.686168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:25.686168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:25.686168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:43:25.686168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:43:25.686168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:43:25.687163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:43:25.687163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:25.687163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:25.687163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:25.687163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:25.687163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:25.687163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:25.687163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:25.687163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:25.687163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:25.687163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:25.687163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:25.687163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:25.687163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:25.687163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:25.687163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:25.687163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:25.688041 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:43:25.688041 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:43:25.688041 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:43:25.688041 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:43:25.688041 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:43:25.688041 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:43:25.688041 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:25.688041 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:25.694810 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 302/600] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 318 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:25.694810 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd360346-b77c-435e-ab76-179498b0f2bf None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:25.699467 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83f0865-4f8f-4c49-8ae4-ce03cb6b0edb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:25.700101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83f0865-4f8f-4c49-8ae4-ce03cb6b0edb None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:25.700381 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83f0865-4f8f-4c49-8ae4-ce03cb6b0edb None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:25.700652 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83f0865-4f8f-4c49-8ae4-ce03cb6b0edb None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:25.701886 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83f0865-4f8f-4c49-8ae4-ce03cb6b0edb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:25.701886 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83f0865-4f8f-4c49-8ae4-ce03cb6b0edb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:25.703172 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83f0865-4f8f-4c49-8ae4-ce03cb6b0edb None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:25.720894 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 299/601] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:43:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:25.720894 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83f0865-4f8f-4c49-8ae4-ce03cb6b0edb None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:25.760646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:25.843529 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83f0865-4f8f-4c49-8ae4-ce03cb6b0edb None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:25.872873 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd360346-b77c-435e-ab76-179498b0f2bf None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:25.884612 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:25.885677 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065cc858-1bd2-4feb-b521-3fd2a90d7725 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:25.888925 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065cc858-1bd2-4feb-b521-3fd2a90d7725 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:25.891287 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065cc858-1bd2-4feb-b521-3fd2a90d7725 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:25.892123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065cc858-1bd2-4feb-b521-3fd2a90d7725 None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:25.896149 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065cc858-1bd2-4feb-b521-3fd2a90d7725 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:25.897102 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065cc858-1bd2-4feb-b521-3fd2a90d7725 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:25.901084 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065cc858-1bd2-4feb-b521-3fd2a90d7725 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:25.922506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 300/602] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:43:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:25.922506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065cc858-1bd2-4feb-b521-3fd2a90d7725 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:25.935599 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98223079-c811-4475-a8fb-93a8a2168565 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:25.937664 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98223079-c811-4475-a8fb-93a8a2168565 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:25.938419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98223079-c811-4475-a8fb-93a8a2168565 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:25.939268 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98223079-c811-4475-a8fb-93a8a2168565 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:25.952132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98223079-c811-4475-a8fb-93a8a2168565 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:25.954981 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98223079-c811-4475-a8fb-93a8a2168565 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:25.974911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-98223079-c811-4475-a8fb-93a8a2168565 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:25.974911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:25.974911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:25.974911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:25.974911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:25.974911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:25.974911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:25.974911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:25.974911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:25.974911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:25.974911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:25.974911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:25.974911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:43:25.974911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:43:25.974911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:43:25.976019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:43:25.976019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:25.976019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:25.976019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:25.976019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:25.976019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:25.976019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:25.976019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:25.976019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:25.976019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:25.976019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:25.976019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:25.976019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:25.976019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:25.976019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:25.976019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:25.976019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:25.977306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:43:25.977306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:43:25.977306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:43:25.977306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:43:25.977306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:43:25.977306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:43:25.977306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:25.977306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:25.986846 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 303/603] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:25.988816 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98223079-c811-4475-a8fb-93a8a2168565 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:26.056388 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:26.079141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065cc858-1bd2-4feb-b521-3fd2a90d7725 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:26.103028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98223079-c811-4475-a8fb-93a8a2168565 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:26.131121 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d72b572-cb7d-4219-b939-7b04cbce6016 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:26.132453 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d72b572-cb7d-4219-b939-7b04cbce6016 None admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:26.132995 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d72b572-cb7d-4219-b939-7b04cbce6016 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:26.133665 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d72b572-cb7d-4219-b939-7b04cbce6016 None admin] PATH_INFO: `/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/a59d73fd45a54651930d52300704fdb0/roles/81523e544c7e4993af4d1e06ce3754bc` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:26.143547 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6653e1b-c151-411d-9c20-dcf9723035a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:26.144372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6653e1b-c151-411d-9c20-dcf9723035a1 None admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:26.144765 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6653e1b-c151-411d-9c20-dcf9723035a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:26.145137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6653e1b-c151-411d-9c20-dcf9723035a1 None admin] PATH_INFO: `/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/9b0510812b60454586a781e6399e77a4/roles/81523e544c7e4993af4d1e06ce3754bc` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:26.156420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d72b572-cb7d-4219-b939-7b04cbce6016 None admin] RBAC: Authorizing `identity:create_grant(project_id=c520a8da3dc54adba4f7cc04ef42e367, user_id=a59d73fd45a54651930d52300704fdb0, role_id=81523e544c7e4993af4d1e06ce3754bc)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:26.158979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d72b572-cb7d-4219-b939-7b04cbce6016 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:26.162487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6653e1b-c151-411d-9c20-dcf9723035a1 None admin] RBAC: Authorizing `identity:create_grant(project_id=c520a8da3dc54adba4f7cc04ef42e367, user_id=9b0510812b60454586a781e6399e77a4, role_id=81523e544c7e4993af4d1e06ce3754bc)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:26.164169 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6653e1b-c151-411d-9c20-dcf9723035a1 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:26.181998 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 301/604] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:43:25 2022] PUT /identity/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/a59d73fd45a54651930d52300704fdb0/roles/81523e544c7e4993af4d1e06ce3754bc => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:43:26.183817 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d72b572-cb7d-4219-b939-7b04cbce6016 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:26.196455 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 304/605] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:43:25 2022] PUT /identity/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/9b0510812b60454586a781e6399e77a4/roles/81523e544c7e4993af4d1e06ce3754bc => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:43:26.381939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d72b572-cb7d-4219-b939-7b04cbce6016 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:26.434982 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be892205-6d52-4144-9a82-7398f0388973 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:26.435861 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be892205-6d52-4144-9a82-7398f0388973 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:26.436189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be892205-6d52-4144-9a82-7398f0388973 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:26.436664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be892205-6d52-4144-9a82-7398f0388973 None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:26.437984 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be892205-6d52-4144-9a82-7398f0388973 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:26.438572 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be892205-6d52-4144-9a82-7398f0388973 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:26.441095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be892205-6d52-4144-9a82-7398f0388973 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:26.445984 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:26.467261 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 302/606] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:43:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:26.473946 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6653e1b-c151-411d-9c20-dcf9723035a1 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:26.497694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:26.600121 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6653e1b-c151-411d-9c20-dcf9723035a1 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:26.642919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e89107b-fd12-4b2f-8f59-061c7f0c7714 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:26.643522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e89107b-fd12-4b2f-8f59-061c7f0c7714 None admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:26.643798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e89107b-fd12-4b2f-8f59-061c7f0c7714 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:26.644072 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e89107b-fd12-4b2f-8f59-061c7f0c7714 None admin] PATH_INFO: `/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/c4c3eaee0a8f4de0b79841a95b35091f/roles/81523e544c7e4993af4d1e06ce3754bc` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:26.662943 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e89107b-fd12-4b2f-8f59-061c7f0c7714 None admin] RBAC: Authorizing `identity:create_grant(project_id=c520a8da3dc54adba4f7cc04ef42e367, user_id=c4c3eaee0a8f4de0b79841a95b35091f, role_id=81523e544c7e4993af4d1e06ce3754bc)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:26.664430 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e89107b-fd12-4b2f-8f59-061c7f0c7714 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:26.682691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 305/607] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:43:26 2022] PUT /identity/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/c4c3eaee0a8f4de0b79841a95b35091f/roles/81523e544c7e4993af4d1e06ce3754bc => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:43:26.829416 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c556a4d-31de-45e2-b25e-cf0ad5b7bffb None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:26.830059 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c556a4d-31de-45e2-b25e-cf0ad5b7bffb None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:26.830663 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c556a4d-31de-45e2-b25e-cf0ad5b7bffb None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:26.833534 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 303/608] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:26 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:26.842948 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f19b04-8340-43c1-9c1e-a361c25f10a1 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:26.843547 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f19b04-8340-43c1-9c1e-a361c25f10a1 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:26.844132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f19b04-8340-43c1-9c1e-a361c25f10a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:26.875723 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-a2f19b04-8340-43c1-9c1e-a361c25f10a1 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:26.877881 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-a2f19b04-8340-43c1-9c1e-a361c25f10a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:26.996101 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:27.029258 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 306/609] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:43:27.460035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c556a4d-31de-45e2-b25e-cf0ad5b7bffb None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:27.578610 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c556a4d-31de-45e2-b25e-cf0ad5b7bffb None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:27.631111 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd488c7-91c4-4ad2-92da-66e5c6141267 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:27.632545 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd488c7-91c4-4ad2-92da-66e5c6141267 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:27.633238 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd488c7-91c4-4ad2-92da-66e5c6141267 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:27.633739 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd488c7-91c4-4ad2-92da-66e5c6141267 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:27.651760 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd488c7-91c4-4ad2-92da-66e5c6141267 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:27.659955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-acd488c7-91c4-4ad2-92da-66e5c6141267 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:27.659955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:27.659955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:27.659955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:27.659955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:27.659955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:27.659955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:27.659955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:27.659955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:27.659955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:27.659955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:27.659955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:27.659955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:27.659955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:27.659955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:27.659955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:27.661578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:27.661578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:27.661578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:27.661578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:27.661578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:27.661578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:27.661578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:27.661578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:27.661578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:27.661578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:27.661578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:27.661578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:27.661578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:27.661578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:27.661578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:27.661578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:27.662758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:27.662758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:27.662758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:27.662758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:27.662758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:27.662758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:27.662758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:27.662758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:27.662758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:27.662758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:27.662758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:27.662758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:27.666501 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 304/610] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:43:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:27.676689 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f19b04-8340-43c1-9c1e-a361c25f10a1 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:27.703503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa316c4-ac3a-4cd6-87e0-7b3e64468177 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:27.704289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa316c4-ac3a-4cd6-87e0-7b3e64468177 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:27.704825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa316c4-ac3a-4cd6-87e0-7b3e64468177 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:27.707608 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 305/611] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:27 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:43:27.713950 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7eee9e-9689-4436-aa7d-1be9813f5947 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:27.714821 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7eee9e-9689-4436-aa7d-1be9813f5947 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:27.715581 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7eee9e-9689-4436-aa7d-1be9813f5947 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:27.766182 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-df7eee9e-9689-4436-aa7d-1be9813f5947 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:27.771148 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-df7eee9e-9689-4436-aa7d-1be9813f5947 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:27.858910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:27.882756 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 306/612] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:43:27.984621 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f19b04-8340-43c1-9c1e-a361c25f10a1 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:28.038773 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7db464-619e-46fd-98d8-01e80ebe1d4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:28.039791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7db464-619e-46fd-98d8-01e80ebe1d4d None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:28.039791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7db464-619e-46fd-98d8-01e80ebe1d4d None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:28.040051 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7db464-619e-46fd-98d8-01e80ebe1d4d None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:28.040571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7db464-619e-46fd-98d8-01e80ebe1d4d None admin] RBAC: Adding query filter params (name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:28.040865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7db464-619e-46fd-98d8-01e80ebe1d4d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:28.042287 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7db464-619e-46fd-98d8-01e80ebe1d4d None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:28.055018 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 307/613] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 18 00:43:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 379 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:28.061544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7eee9e-9689-4436-aa7d-1be9813f5947 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:28.187022 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7db464-619e-46fd-98d8-01e80ebe1d4d None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:28.241467 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7eee9e-9689-4436-aa7d-1be9813f5947 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:28.281990 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a51badb-0c3f-427e-abb3-73db31b6f5dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:28.283069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a51badb-0c3f-427e-abb3-73db31b6f5dc None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:28.283547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a51badb-0c3f-427e-abb3-73db31b6f5dc None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:28.284308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a51badb-0c3f-427e-abb3-73db31b6f5dc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:28.297793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a51badb-0c3f-427e-abb3-73db31b6f5dc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:28.299801 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a51badb-0c3f-427e-abb3-73db31b6f5dc None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:28.300443 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:28.318826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-9a51badb-0c3f-427e-abb3-73db31b6f5dc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:28.318826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:28.318826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:28.318826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:28.318826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:28.318826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:28.318826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:28.318826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:28.318826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:28.318826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:28.318826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:28.318826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:28.318826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:28.318826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:28.318826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:28.318826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:28.319837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:28.319837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:28.319837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:28.319837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:28.319837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:28.319837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:28.319837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:28.319837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:28.319837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:28.319837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:28.319837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:28.319837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:28.319837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:28.319837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:28.319837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:28.319837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:28.319837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:28.320709 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:28.320709 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:28.320709 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:28.323798 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 307/614] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:28.333709 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a51badb-0c3f-427e-abb3-73db31b6f5dc None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:28.420519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7db464-619e-46fd-98d8-01e80ebe1d4d None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:28.422822 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:28.492181 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecfe5f7-e2b9-44c9-84c8-7a18872c5948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:28.492739 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecfe5f7-e2b9-44c9-84c8-7a18872c5948 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:28.493016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecfe5f7-e2b9-44c9-84c8-7a18872c5948 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:28.493286 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecfe5f7-e2b9-44c9-84c8-7a18872c5948 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:28.524943 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecfe5f7-e2b9-44c9-84c8-7a18872c5948 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:28.531766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-fecfe5f7-e2b9-44c9-84c8-7a18872c5948 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:28.531766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:28.531766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:28.531766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:28.531766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:28.531766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:28.531766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:28.531766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:28.531766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:28.531766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:28.531766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:28.531766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:28.531766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:28.531766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:28.531766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:28.531766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:28.532736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:28.532736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:28.532736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:28.532736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:28.532736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:28.532736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:28.532736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:28.532736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:28.532736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:28.532736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:28.532736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:28.532736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:28.532736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:28.532736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:28.532736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:28.532736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:28.533560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:28.533560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:28.533560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:28.533560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:28.533560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:28.533560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:28.533560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:28.533560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:28.533560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:28.533560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:28.533560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:28.533560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:28.539545 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 308/615] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:43:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 353 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:28.546987 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecfe5f7-e2b9-44c9-84c8-7a18872c5948 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:28.559138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a51badb-0c3f-427e-abb3-73db31b6f5dc None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:28.661195 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8afdfd1-feb7-4236-9929-eb31b63732ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:28.662240 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8afdfd1-feb7-4236-9929-eb31b63732ca None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:28.662745 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8afdfd1-feb7-4236-9929-eb31b63732ca None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:28.663218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8afdfd1-feb7-4236-9929-eb31b63732ca None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:28.664090 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8afdfd1-feb7-4236-9929-eb31b63732ca None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:28.664595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8afdfd1-feb7-4236-9929-eb31b63732ca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:28.668271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8afdfd1-feb7-4236-9929-eb31b63732ca None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:28.696799 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecfe5f7-e2b9-44c9-84c8-7a18872c5948 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:28.712819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 308/616] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 379 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:28.719534 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8afdfd1-feb7-4236-9929-eb31b63732ca None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:28.728807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:28.748784 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5b78f0-0149-4ed5-8d8d-839bcd9889a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:28.749452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5b78f0-0149-4ed5-8d8d-839bcd9889a7 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:28.749834 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5b78f0-0149-4ed5-8d8d-839bcd9889a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:28.750210 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5b78f0-0149-4ed5-8d8d-839bcd9889a7 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:28.751029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5b78f0-0149-4ed5-8d8d-839bcd9889a7 None admin] RBAC: Adding query filter params (name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:28.751435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5b78f0-0149-4ed5-8d8d-839bcd9889a7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:28.752916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5b78f0-0149-4ed5-8d8d-839bcd9889a7 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:28.782505 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 309/617] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 18 00:43:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:43:28.793939 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5b78f0-0149-4ed5-8d8d-839bcd9889a7 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:28.916051 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:28.924568 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8afdfd1-feb7-4236-9929-eb31b63732ca None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:28.930546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5b78f0-0149-4ed5-8d8d-839bcd9889a7 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:28.975028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411f5d01-d6fc-4db4-81c6-cdd3c03fb3be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:28.976399 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411f5d01-d6fc-4db4-81c6-cdd3c03fb3be None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:28.976873 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411f5d01-d6fc-4db4-81c6-cdd3c03fb3be None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:28.977159 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411f5d01-d6fc-4db4-81c6-cdd3c03fb3be None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:28.998719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760ae91b-9ecc-4902-b120-ca2c6b5f0fee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:28.999429 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760ae91b-9ecc-4902-b120-ca2c6b5f0fee None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:28.999650 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760ae91b-9ecc-4902-b120-ca2c6b5f0fee None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:28.999926 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760ae91b-9ecc-4902-b120-ca2c6b5f0fee None admin] PATH_INFO: `/v3/users/swift` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:29.001337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411f5d01-d6fc-4db4-81c6-cdd3c03fb3be None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:29.004443 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411f5d01-d6fc-4db4-81c6-cdd3c03fb3be None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:29.017223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760ae91b-9ecc-4902-b120-ca2c6b5f0fee None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:29.018729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760ae91b-9ecc-4902-b120-ca2c6b5f0fee None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:29.035595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-411f5d01-d6fc-4db4-81c6-cdd3c03fb3be None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:29.035595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:29.035595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:29.035595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:29.035595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:29.035595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:29.035595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:29.035595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:29.035595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:29.035595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:29.035595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:29.035595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:29.035595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:29.035595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:29.035595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:29.035595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:29.036730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:29.036730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:29.036730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:29.036730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:29.036730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:29.036730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:29.036730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:29.036730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:29.036730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:29.036730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:29.036730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:29.036730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:29.036730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:29.036730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:29.036730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:29.036730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:29.036730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:29.038123 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:29.038123 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:29.038123 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:29.038123 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 310/618] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:29.043413 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411f5d01-d6fc-4db4-81c6-cdd3c03fb3be None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:29.046678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-760ae91b-9ecc-4902-b120-ca2c6b5f0fee None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 18 00:43:29.046678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:29.046678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:29.046678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:29.046678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:29.046678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:29.046678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:29.046678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:29.046678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:29.046678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:29.046678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:29.046678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:29.046678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:43:29.046678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:43:29.046678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:43:29.046678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:43:29.048858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:29.048858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:29.048858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:43:29.048858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:29.048858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:43:29.048858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:29.048858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:29.048858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:29.048858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:29.048858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:29.048858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:29.048858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:29.048858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:29.048858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:29.048858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:29.048858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:29.048858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:29.048858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:29.051449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:29.051449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:29.051449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:43:29.051449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:43:29.051449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:43:29.051449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:43:29.051449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:43:29.051449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:43:29.051449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 18 00:43:29.051449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:29.051449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 309/619] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:43:28 2022] GET /identity/v3/users/swift => generated 83 bytes in 329 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:29.058959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760ae91b-9ecc-4902-b120-ca2c6b5f0fee None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:29.085671 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:29.170974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411f5d01-d6fc-4db4-81c6-cdd3c03fb3be None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:29.220528 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e089f24-b2ba-4d45-8bda-08bf0ed21ab5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:29.221213 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e089f24-b2ba-4d45-8bda-08bf0ed21ab5 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:29.221489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e089f24-b2ba-4d45-8bda-08bf0ed21ab5 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:29.221757 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e089f24-b2ba-4d45-8bda-08bf0ed21ab5 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:29.222494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e089f24-b2ba-4d45-8bda-08bf0ed21ab5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:29.222807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e089f24-b2ba-4d45-8bda-08bf0ed21ab5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:29.224212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e089f24-b2ba-4d45-8bda-08bf0ed21ab5 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:29.238505 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 311/620] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:29.239859 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ef811d-e766-4ae8-9c44-8e3812a675f2 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:29.240307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ef811d-e766-4ae8-9c44-8e3812a675f2 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:29.240615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ef811d-e766-4ae8-9c44-8e3812a675f2 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:29.242288 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 312/621] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:29.248247 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91693351-91dc-4445-b74b-4616cfbd47d2 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:29.248539 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91693351-91dc-4445-b74b-4616cfbd47d2 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:29.248790 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91693351-91dc-4445-b74b-4616cfbd47d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:29.269063 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-91693351-91dc-4445-b74b-4616cfbd47d2 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:29.270193 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-91693351-91dc-4445-b74b-4616cfbd47d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:29.285739 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760ae91b-9ecc-4902-b120-ca2c6b5f0fee None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:29.330237 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7f374c-27d2-4a8a-bd1d-df84a8c1e1a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:29.330893 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7f374c-27d2-4a8a-bd1d-df84a8c1e1a5 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:29.331175 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7f374c-27d2-4a8a-bd1d-df84a8c1e1a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:29.331452 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7f374c-27d2-4a8a-bd1d-df84a8c1e1a5 None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:29.332203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7f374c-27d2-4a8a-bd1d-df84a8c1e1a5 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:29.332513 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7f374c-27d2-4a8a-bd1d-df84a8c1e1a5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:29.334312 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7f374c-27d2-4a8a-bd1d-df84a8c1e1a5 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:29.362717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 310/622] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Sep 18 00:43:29 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:43:29.366307 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7f374c-27d2-4a8a-bd1d-df84a8c1e1a5 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:29.403779 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:29.434976 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 313/623] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:43:29.436711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91693351-91dc-4445-b74b-4616cfbd47d2 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:29.535011 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7f374c-27d2-4a8a-bd1d-df84a8c1e1a5 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:29.555265 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:29.592074 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f140f8c8-08f0-4cc7-bb6e-1a4628b4b33c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:29.594205 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f140f8c8-08f0-4cc7-bb6e-1a4628b4b33c None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:29.594951 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f140f8c8-08f0-4cc7-bb6e-1a4628b4b33c None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:29.595625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f140f8c8-08f0-4cc7-bb6e-1a4628b4b33c None admin] PATH_INFO: `/v3/users/glance` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:29.614774 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f140f8c8-08f0-4cc7-bb6e-1a4628b4b33c None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:29.619743 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f140f8c8-08f0-4cc7-bb6e-1a4628b4b33c None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:29.652160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-f140f8c8-08f0-4cc7-bb6e-1a4628b4b33c None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 18 00:43:29.652160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:29.652160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:29.652160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:29.652160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:29.652160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:29.652160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:29.652160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:29.652160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:29.652160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:29.652160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:29.652160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:29.652160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:43:29.652160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:43:29.652160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:43:29.652160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:43:29.653203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:29.653203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:29.653203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:43:29.653203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:29.653203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:43:29.653203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:29.653203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:29.653203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:29.653203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:29.653203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:29.653203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:29.653203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:29.653203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:29.653203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:29.653203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:29.653203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:29.653203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:29.653203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:29.654146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:29.654146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:29.654146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:43:29.654146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:43:29.654146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:43:29.654146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:43:29.654146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:43:29.654146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:43:29.654146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 18 00:43:29.654146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:29.659594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 311/624] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:43:29 2022] GET /identity/v3/users/glance => generated 84 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:29.661232 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6028ad-46c8-45f9-970d-f849f1cd6df2 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:29.663510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6028ad-46c8-45f9-970d-f849f1cd6df2 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:29.665328 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6028ad-46c8-45f9-970d-f849f1cd6df2 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:29.669114 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 312/625] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:29 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:29.671911 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6028ad-46c8-45f9-970d-f849f1cd6df2 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:29.750441 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91693351-91dc-4445-b74b-4616cfbd47d2 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:29.799021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:29.804045 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16878445-7768-4219-8d80-1cb86f270c03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:29.804512 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16878445-7768-4219-8d80-1cb86f270c03 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:29.804767 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16878445-7768-4219-8d80-1cb86f270c03 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:29.805036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16878445-7768-4219-8d80-1cb86f270c03 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:29.827127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16878445-7768-4219-8d80-1cb86f270c03 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:29.828619 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16878445-7768-4219-8d80-1cb86f270c03 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:29.834725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6028ad-46c8-45f9-970d-f849f1cd6df2 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:29.851310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-16878445-7768-4219-8d80-1cb86f270c03 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:29.851310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:29.851310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:29.851310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:29.851310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:29.851310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:29.851310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:29.851310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:29.851310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:29.851310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:29.851310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:29.851310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:29.851310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:29.851310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:29.851310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:29.851310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:29.852307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:29.852307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:29.852307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:29.852307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:29.852307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:29.852307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:29.852307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:29.852307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:29.852307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:29.852307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:29.852307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:29.852307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:29.852307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:29.852307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:29.852307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:29.852307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:29.852307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:29.853236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:29.853236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:29.853236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:29.853854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 314/626] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 418 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:29.855131 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5243ba94-0f8f-49f5-b493-973534387b62 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:29.855418 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5243ba94-0f8f-49f5-b493-973534387b62 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:29.855694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5243ba94-0f8f-49f5-b493-973534387b62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:29.880378 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-5243ba94-0f8f-49f5-b493-973534387b62 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:29.882259 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-5243ba94-0f8f-49f5-b493-973534387b62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:29.895624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c061a439-e699-4395-84f9-d7910e0253aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:29.896520 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c061a439-e699-4395-84f9-d7910e0253aa None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:29.896764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c061a439-e699-4395-84f9-d7910e0253aa None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:29.897143 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c061a439-e699-4395-84f9-d7910e0253aa None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:29.898101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c061a439-e699-4395-84f9-d7910e0253aa None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:29.898533 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c061a439-e699-4395-84f9-d7910e0253aa None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:29.900775 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c061a439-e699-4395-84f9-d7910e0253aa None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:29.928618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 313/627] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Sep 18 00:43:29 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:29.930013 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b7cc1c-5e68-41a4-978b-be62d7f106ee None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:29.930287 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b7cc1c-5e68-41a4-978b-be62d7f106ee None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:29.930661 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b7cc1c-5e68-41a4-978b-be62d7f106ee None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:29.933621 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 314/628] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:29.933621 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b7cc1c-5e68-41a4-978b-be62d7f106ee None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:29.991301 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 315/629] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:43:29.993020 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5243ba94-0f8f-49f5-b493-973534387b62 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:30.004701 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:30.068014 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b7cc1c-5e68-41a4-978b-be62d7f106ee None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:30.116127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5243ba94-0f8f-49f5-b493-973534387b62 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:30.127460 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31802708-b301-4fb0-bc57-0447e9b67b3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:30.128820 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31802708-b301-4fb0-bc57-0447e9b67b3d None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:30.129510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31802708-b301-4fb0-bc57-0447e9b67b3d None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:30.130195 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31802708-b301-4fb0-bc57-0447e9b67b3d None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:30.131603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31802708-b301-4fb0-bc57-0447e9b67b3d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:30.132368 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31802708-b301-4fb0-bc57-0447e9b67b3d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:30.134849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31802708-b301-4fb0-bc57-0447e9b67b3d None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:30.138351 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:30.157293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d00944-20c8-4cef-befa-47e05c2494c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:30.157965 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d00944-20c8-4cef-befa-47e05c2494c7 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:30.158253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d00944-20c8-4cef-befa-47e05c2494c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:30.158543 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d00944-20c8-4cef-befa-47e05c2494c7 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:30.163129 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 315/630] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:30.163582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31802708-b301-4fb0-bc57-0447e9b67b3d None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:30.174908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d00944-20c8-4cef-befa-47e05c2494c7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:30.180077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-17d00944-20c8-4cef-befa-47e05c2494c7 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:30.180077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:30.180077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:30.180077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:30.180077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:30.180077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:30.180077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:30.180077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:30.180077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:30.180077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:30.180077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:30.180077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:30.180077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:30.180077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:30.180077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:30.180077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:30.187212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:30.187212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:30.187212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:30.187212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:30.187212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:30.187212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:30.187212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:30.187212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:30.187212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:30.187212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:30.187212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:30.187212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:30.187212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:30.187212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:30.187212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:30.187212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:30.188049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:30.188049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:30.188049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:30.188049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:30.188049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:30.188049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:30.188049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:30.188049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:30.188049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:30.188049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:30.188049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:30.188049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:30.188049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 316/631] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:43:29 2022] GET /identity/v3/roles/service => generated 85 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:30.188049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dda25b-7ac4-4f64-b4f5-1d4eda456b5a None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:30.188820 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dda25b-7ac4-4f64-b4f5-1d4eda456b5a None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:30.188820 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dda25b-7ac4-4f64-b4f5-1d4eda456b5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:30.206985 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-a7dda25b-7ac4-4f64-b4f5-1d4eda456b5a None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:30.208330 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-a7dda25b-7ac4-4f64-b4f5-1d4eda456b5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:30.280509 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:30.291779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 317/632] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:43:30.293465 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dda25b-7ac4-4f64-b4f5-1d4eda456b5a None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:30.360552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31802708-b301-4fb0-bc57-0447e9b67b3d None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:30.415738 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749abbab-753b-4740-a110-e6027ee64ec1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:30.416373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749abbab-753b-4740-a110-e6027ee64ec1 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:30.416671 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749abbab-753b-4740-a110-e6027ee64ec1 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:30.416917 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749abbab-753b-4740-a110-e6027ee64ec1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:30.451476 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dda25b-7ac4-4f64-b4f5-1d4eda456b5a None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:30.503847 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ab2dd5-e26c-4deb-8f01-7ef6e2b21f91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:30.504495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ab2dd5-e26c-4deb-8f01-7ef6e2b21f91 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:30.505301 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ab2dd5-e26c-4deb-8f01-7ef6e2b21f91 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:30.505469 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ab2dd5-e26c-4deb-8f01-7ef6e2b21f91 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:30.517649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:30.519803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ab2dd5-e26c-4deb-8f01-7ef6e2b21f91 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:30.521334 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ab2dd5-e26c-4deb-8f01-7ef6e2b21f91 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:30.542779 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749abbab-753b-4740-a110-e6027ee64ec1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:30.543015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-93ab2dd5-e26c-4deb-8f01-7ef6e2b21f91 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:30.543015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:30.543015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:30.543015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:30.543015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:30.543015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:30.543015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:30.543015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:30.543015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:30.543015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:30.543015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:30.543015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:30.543015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:43:30.543015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:43:30.543015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:43:30.543841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:43:30.543841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:30.543841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:30.543841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:30.543841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:30.543841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:30.543841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:30.543841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:30.543841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:30.543841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:30.543841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:30.543841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:30.543841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:30.543841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:30.543841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:30.543841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:30.543841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:30.544706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:43:30.544706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:43:30.544706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:43:30.544706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:43:30.544706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:43:30.544706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:43:30.544706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:30.544706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:30.545605 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 318/633] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:30.546197 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749abbab-753b-4740-a110-e6027ee64ec1 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:30.546737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ab2dd5-e26c-4deb-8f01-7ef6e2b21f91 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:30.619260 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:30.671228 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ab2dd5-e26c-4deb-8f01-7ef6e2b21f91 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:30.681614 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 316/634] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:43:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 518 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:43:30.682503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749abbab-753b-4740-a110-e6027ee64ec1 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:30.714114 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f129d7e2-b8af-4b8b-91a3-5f98c3683cba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:30.714763 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f129d7e2-b8af-4b8b-91a3-5f98c3683cba None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:30.715043 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f129d7e2-b8af-4b8b-91a3-5f98c3683cba None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:30.715315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f129d7e2-b8af-4b8b-91a3-5f98c3683cba None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:30.715990 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f129d7e2-b8af-4b8b-91a3-5f98c3683cba None admin] RBAC: Adding query filter params (name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:30.716291 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f129d7e2-b8af-4b8b-91a3-5f98c3683cba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:30.717666 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f129d7e2-b8af-4b8b-91a3-5f98c3683cba None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:30.727353 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 319/635] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 18 00:43:30 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:30.727353 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f129d7e2-b8af-4b8b-91a3-5f98c3683cba None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:30.812691 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:30.822662 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749abbab-753b-4740-a110-e6027ee64ec1 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:30.851029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f129d7e2-b8af-4b8b-91a3-5f98c3683cba None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:30.866840 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2d1a47-1740-4e1f-9d24-baf3df4865a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:30.867838 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2d1a47-1740-4e1f-9d24-baf3df4865a4 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:30.868133 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2d1a47-1740-4e1f-9d24-baf3df4865a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:30.868442 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2d1a47-1740-4e1f-9d24-baf3df4865a4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:30.872090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:30.883166 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2d1a47-1740-4e1f-9d24-baf3df4865a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:30.888687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-ed2d1a47-1740-4e1f-9d24-baf3df4865a4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:30.888687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:30.888687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:30.888687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:30.888687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:30.888687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:30.888687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:30.888687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:30.888687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:30.888687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:30.888687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:30.888687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:30.888687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:30.888687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:30.888687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:30.888687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:30.889715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:30.889715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:30.889715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:30.889715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:30.889715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:30.889715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:30.889715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:30.889715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:30.889715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:30.889715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:30.889715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:30.889715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:30.889715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:30.889715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:30.889715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:30.889715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:30.890585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:30.890585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:30.890585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:30.890585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:30.890585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:30.890585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:30.890585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:30.890585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:30.890585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:30.890585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:30.890585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:30.890585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:30.890585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 317/636] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:43:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:30.891669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2d1a47-1740-4e1f-9d24-baf3df4865a4 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:30.893510 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fd86bf-beee-4b7c-91bd-beca975fc66a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:30.894136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fd86bf-beee-4b7c-91bd-beca975fc66a None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:30.894417 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fd86bf-beee-4b7c-91bd-beca975fc66a None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:30.895124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fd86bf-beee-4b7c-91bd-beca975fc66a None admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:30.895591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fd86bf-beee-4b7c-91bd-beca975fc66a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:30.895901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fd86bf-beee-4b7c-91bd-beca975fc66a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:30.897358 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fd86bf-beee-4b7c-91bd-beca975fc66a None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:30.910107 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 320/637] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:43:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:30.910905 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fd86bf-beee-4b7c-91bd-beca975fc66a None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:31.029039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2d1a47-1740-4e1f-9d24-baf3df4865a4 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:31.045389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fd86bf-beee-4b7c-91bd-beca975fc66a None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:31.076923 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778a5879-097d-45e9-b442-5b9a1369cced None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:31.077577 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778a5879-097d-45e9-b442-5b9a1369cced None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:31.077859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778a5879-097d-45e9-b442-5b9a1369cced None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:31.078133 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778a5879-097d-45e9-b442-5b9a1369cced None admin] PATH_INFO: `/v3/roles/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:31.084820 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f660778-afbd-4434-a7ca-a11b1e6117f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:31.085473 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f660778-afbd-4434-a7ca-a11b1e6117f5 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:31.085754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f660778-afbd-4434-a7ca-a11b1e6117f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:31.086029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f660778-afbd-4434-a7ca-a11b1e6117f5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:31.090733 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-778a5879-097d-45e9-b442-5b9a1369cced None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:31.095905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-778a5879-097d-45e9-b442-5b9a1369cced None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:31.095905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:31.095905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:31.095905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:31.095905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:31.095905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:31.095905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:31.095905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:31.095905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:31.095905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:31.095905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:31.095905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:31.095905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:31.095905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:31.095905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:31.095905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:31.097666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:31.097666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:31.097666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:31.097666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:31.097666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:31.097666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:31.097666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:31.097666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:31.097666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:31.097666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:31.097666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:31.097666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:31.097666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:31.097666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:31.097666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:31.097666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:31.099199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:31.099199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:31.099199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:31.099199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:31.099199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:31.099199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:31.099199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:31.099199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:31.099199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:31.099199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:31.099199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:31.099199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:31.099199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 318/638] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:43:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:31.099199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778a5879-097d-45e9-b442-5b9a1369cced None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:31.100702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f660778-afbd-4434-a7ca-a11b1e6117f5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:31.101590 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f660778-afbd-4434-a7ca-a11b1e6117f5 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:31.117607 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:31.117993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-1f660778-afbd-4434-a7ca-a11b1e6117f5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:31.117993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:31.117993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:31.117993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:31.117993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:31.117993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:31.117993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:31.117993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:31.117993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:31.117993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:31.117993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:31.117993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:31.117993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:31.117993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:31.117993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:31.117993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:31.121036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:31.121036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:31.121036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:31.121036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:31.121036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:31.121036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:31.121036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:31.121036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:31.121036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:31.121036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:31.121036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:31.121036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:31.121036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:31.121036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:31.121036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:31.121036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:31.121036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:31.123224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:31.123224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:31.123224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:31.123224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 321/639] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:31.123224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f660778-afbd-4434-a7ca-a11b1e6117f5 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:31.158316 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:31.228854 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778a5879-097d-45e9-b442-5b9a1369cced None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:31.249868 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f660778-afbd-4434-a7ca-a11b1e6117f5 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:31.275515 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4933e237-38f6-43ef-976e-5eaf78175741 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:31.276186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4933e237-38f6-43ef-976e-5eaf78175741 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:31.276467 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4933e237-38f6-43ef-976e-5eaf78175741 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:31.277286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4933e237-38f6-43ef-976e-5eaf78175741 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:31.289663 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4933e237-38f6-43ef-976e-5eaf78175741 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:31.291721 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4933e237-38f6-43ef-976e-5eaf78175741 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:31.294780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9216cdbf-3487-48c4-a119-b3dbc6e98917 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:31.295832 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9216cdbf-3487-48c4-a119-b3dbc6e98917 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:31.296300 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9216cdbf-3487-48c4-a119-b3dbc6e98917 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:31.296798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9216cdbf-3487-48c4-a119-b3dbc6e98917 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:31.298067 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9216cdbf-3487-48c4-a119-b3dbc6e98917 None admin] RBAC: Adding query filter params (name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:31.298648 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9216cdbf-3487-48c4-a119-b3dbc6e98917 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:31.301159 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9216cdbf-3487-48c4-a119-b3dbc6e98917 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:31.310432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-4933e237-38f6-43ef-976e-5eaf78175741 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:31.310432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:31.310432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:31.310432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:31.310432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:31.310432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:31.310432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:31.310432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:31.310432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:31.310432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:31.310432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:31.310432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:31.310432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:31.310432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:31.310432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:31.310432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:31.311727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:31.311727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:31.311727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:31.311727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:31.311727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:31.311727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:31.311727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:31.311727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:31.311727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:31.311727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:31.311727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:31.311727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:31.311727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:31.311727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:31.311727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:31.311727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:31.311727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:31.313208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:31.313208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:31.313208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:31.313208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 319/640] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:31.313640 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 322/641] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 18 00:43:31 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:31.313640 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9216cdbf-3487-48c4-a119-b3dbc6e98917 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:31.315086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4933e237-38f6-43ef-976e-5eaf78175741 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:31.406425 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:31.417811 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:31.429026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9216cdbf-3487-48c4-a119-b3dbc6e98917 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:31.460378 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4933e237-38f6-43ef-976e-5eaf78175741 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:31.475231 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12758cd1-c734-4789-9c32-058f62ad6801 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:31.476056 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12758cd1-c734-4789-9c32-058f62ad6801 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:31.476446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12758cd1-c734-4789-9c32-058f62ad6801 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:31.476903 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12758cd1-c734-4789-9c32-058f62ad6801 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:31.477720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12758cd1-c734-4789-9c32-058f62ad6801 None admin] RBAC: Adding query filter params (name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:31.478222 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12758cd1-c734-4789-9c32-058f62ad6801 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:31.480509 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12758cd1-c734-4789-9c32-058f62ad6801 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:31.495990 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 323/642] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 18 00:43:31 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:31.495990 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12758cd1-c734-4789-9c32-058f62ad6801 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:31.501689 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d773d8b0-5636-4184-88ba-26ccaa54a5f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:31.502358 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d773d8b0-5636-4184-88ba-26ccaa54a5f0 None admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:31.502670 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d773d8b0-5636-4184-88ba-26ccaa54a5f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:31.502948 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d773d8b0-5636-4184-88ba-26ccaa54a5f0 None admin] PATH_INFO: `/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/6230ed7ce0eb43e9b2b8530a28eb6299/roles/81523e544c7e4993af4d1e06ce3754bc` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:31.522656 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d773d8b0-5636-4184-88ba-26ccaa54a5f0 None admin] RBAC: Authorizing `identity:create_grant(project_id=c520a8da3dc54adba4f7cc04ef42e367, user_id=6230ed7ce0eb43e9b2b8530a28eb6299, role_id=81523e544c7e4993af4d1e06ce3754bc)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:31.524116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d773d8b0-5636-4184-88ba-26ccaa54a5f0 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:31.544655 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 320/643] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:43:31 2022] PUT /identity/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/6230ed7ce0eb43e9b2b8530a28eb6299/roles/81523e544c7e4993af4d1e06ce3754bc => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:43:31.546115 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d773d8b0-5636-4184-88ba-26ccaa54a5f0 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:31.638298 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12758cd1-c734-4789-9c32-058f62ad6801 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:31.706678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d51fc5-4931-4e38-8576-3b83e36578fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:31.707831 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d51fc5-4931-4e38-8576-3b83e36578fd None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:31.708453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d51fc5-4931-4e38-8576-3b83e36578fd None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:31.708989 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d51fc5-4931-4e38-8576-3b83e36578fd None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:31.710225 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d51fc5-4931-4e38-8576-3b83e36578fd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:31.710889 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d51fc5-4931-4e38-8576-3b83e36578fd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:31.713391 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d51fc5-4931-4e38-8576-3b83e36578fd None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:31.736778 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 324/644] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:31.738717 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d51fc5-4931-4e38-8576-3b83e36578fd None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:31.748231 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d773d8b0-5636-4184-88ba-26ccaa54a5f0 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:31.768631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:31.781624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:31.802559 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c212d794-ab2a-4c68-bb75-9c56f8c4585c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:31.803256 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c212d794-ab2a-4c68-bb75-9c56f8c4585c None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:31.803550 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c212d794-ab2a-4c68-bb75-9c56f8c4585c None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:31.803825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c212d794-ab2a-4c68-bb75-9c56f8c4585c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:31.817506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c212d794-ab2a-4c68-bb75-9c56f8c4585c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:31.819072 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c212d794-ab2a-4c68-bb75-9c56f8c4585c None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:31.847903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-c212d794-ab2a-4c68-bb75-9c56f8c4585c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:31.847903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:31.847903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:31.847903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:31.847903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:31.847903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:31.847903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:31.847903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:31.847903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:31.847903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:31.847903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:31.847903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:31.847903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:31.847903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:31.847903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:31.847903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:31.857763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:31.857763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:31.857763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:31.857763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:31.857763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:31.857763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:31.857763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:31.857763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:31.857763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:31.857763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:31.857763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:31.857763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:31.857763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:31.857763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:31.857763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:31.857763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:31.857763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:31.859838 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:31.859838 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:31.859838 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:31.859838 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 321/645] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 304 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:31.859838 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c212d794-ab2a-4c68-bb75-9c56f8c4585c None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:31.934242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d51fc5-4931-4e38-8576-3b83e36578fd None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:31.997674 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194d25f2-ecd6-40eb-ae95-133f4dba07f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:31.998574 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194d25f2-ecd6-40eb-ae95-133f4dba07f8 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:31.999018 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194d25f2-ecd6-40eb-ae95-133f4dba07f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:31.999566 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194d25f2-ecd6-40eb-ae95-133f4dba07f8 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:32.000712 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194d25f2-ecd6-40eb-ae95-133f4dba07f8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:32.001149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194d25f2-ecd6-40eb-ae95-133f4dba07f8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:32.002792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194d25f2-ecd6-40eb-ae95-133f4dba07f8 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:32.005821 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c212d794-ab2a-4c68-bb75-9c56f8c4585c None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:32.027067 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 325/646] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:32.027067 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194d25f2-ecd6-40eb-ae95-133f4dba07f8 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:32.070924 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef82183-8066-4202-b73b-253e8954a844 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:32.071984 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef82183-8066-4202-b73b-253e8954a844 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:32.072479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef82183-8066-4202-b73b-253e8954a844 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:32.072966 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef82183-8066-4202-b73b-253e8954a844 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:32.094494 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef82183-8066-4202-b73b-253e8954a844 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:32.097300 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef82183-8066-4202-b73b-253e8954a844 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:32.120812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-8ef82183-8066-4202-b73b-253e8954a844 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:32.120812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:32.120812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:32.120812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:32.120812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:32.120812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:32.120812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:32.120812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:32.120812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:32.120812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:32.120812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:32.120812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:32.120812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:32.120812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:32.120812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:32.120812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:32.122667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:32.122667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:32.122667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:32.122667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:32.122667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:32.122667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:32.122667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:32.122667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:32.122667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:32.122667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:32.122667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:32.122667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:32.122667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:32.122667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:32.122667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:32.122667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:32.122667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:32.125387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:32.125387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:32.125387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:32.125387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 322/647] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:32.125387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef82183-8066-4202-b73b-253e8954a844 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:32.145122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:32.165278 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:32.181619 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194d25f2-ecd6-40eb-ae95-133f4dba07f8 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:32.243119 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48ce0e0-d936-4b06-ac00-38f754a8fd9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:32.244035 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48ce0e0-d936-4b06-ac00-38f754a8fd9a None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:32.244471 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48ce0e0-d936-4b06-ac00-38f754a8fd9a None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:32.244921 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48ce0e0-d936-4b06-ac00-38f754a8fd9a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:32.259590 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48ce0e0-d936-4b06-ac00-38f754a8fd9a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:32.262662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48ce0e0-d936-4b06-ac00-38f754a8fd9a None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:32.284026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-c48ce0e0-d936-4b06-ac00-38f754a8fd9a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:32.284026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:32.284026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:32.284026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:32.284026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:32.284026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:32.284026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:32.284026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:32.284026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:32.284026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:32.284026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:32.284026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:32.284026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:43:32.284026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:43:32.284026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:43:32.285766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:43:32.285766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:32.285766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:32.285766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:32.285766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:32.285766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:32.285766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:32.285766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:32.285766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:32.285766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:32.285766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:32.285766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:32.285766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:32.285766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:32.285766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:32.285766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:32.285766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:32.288638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:43:32.288638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:43:32.288638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:43:32.288638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:43:32.288638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:43:32.288638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:43:32.288638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:32.288638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:32.288638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 326/648] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:32.288638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48ce0e0-d936-4b06-ac00-38f754a8fd9a None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:32.323221 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef82183-8066-4202-b73b-253e8954a844 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:32.382708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9919c6a4-d3a8-46ec-8e07-f5750a4b25e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:32.383542 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9919c6a4-d3a8-46ec-8e07-f5750a4b25e5 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:32.383831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9919c6a4-d3a8-46ec-8e07-f5750a4b25e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:32.384104 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9919c6a4-d3a8-46ec-8e07-f5750a4b25e5 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:32.385030 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9919c6a4-d3a8-46ec-8e07-f5750a4b25e5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:32.385518 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9919c6a4-d3a8-46ec-8e07-f5750a4b25e5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:32.388194 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9919c6a4-d3a8-46ec-8e07-f5750a4b25e5 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:32.409544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 323/649] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:32.411059 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9919c6a4-d3a8-46ec-8e07-f5750a4b25e5 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:32.479435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48ce0e0-d936-4b06-ac00-38f754a8fd9a None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:32.540261 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:32.550099 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2adb2ec-2752-446c-bf36-e21e9a2578f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:32.551522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2adb2ec-2752-446c-bf36-e21e9a2578f0 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:32.551878 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2adb2ec-2752-446c-bf36-e21e9a2578f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:32.552285 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2adb2ec-2752-446c-bf36-e21e9a2578f0 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:32.561123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:32.567102 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2adb2ec-2752-446c-bf36-e21e9a2578f0 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:32.569897 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2adb2ec-2752-446c-bf36-e21e9a2578f0 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:32.586830 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9919c6a4-d3a8-46ec-8e07-f5750a4b25e5 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:32.592422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-c2adb2ec-2752-446c-bf36-e21e9a2578f0 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 18 00:43:32.592422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:32.592422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:32.592422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:32.592422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:32.592422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:32.592422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:32.592422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:32.592422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:32.592422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:32.592422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:32.592422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:32.592422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:43:32.592422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:43:32.592422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:43:32.592422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:43:32.594302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:32.594302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:32.594302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:43:32.594302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:32.594302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:43:32.594302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:32.594302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:32.594302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:32.594302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:32.594302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:32.594302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:32.594302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:32.594302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:32.594302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:32.594302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:32.594302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:32.594302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:32.594302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:32.596632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:32.596632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:32.596632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:43:32.596632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:43:32.596632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:43:32.596632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:43:32.596632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:43:32.596632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:43:32.596632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 18 00:43:32.596632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:32.601687 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 327/650] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Sep 18 00:43:32 2022] GET /identity/v3/users/nova => generated 82 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:32.603459 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2adb2ec-2752-446c-bf36-e21e9a2578f0 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:32.634907 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc755c02-d38c-4a48-858f-c610b040a91a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:32.635708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc755c02-d38c-4a48-858f-c610b040a91a None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:32.636370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc755c02-d38c-4a48-858f-c610b040a91a None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:32.636805 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc755c02-d38c-4a48-858f-c610b040a91a None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:32.637650 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc755c02-d38c-4a48-858f-c610b040a91a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:32.638095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc755c02-d38c-4a48-858f-c610b040a91a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:32.639773 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc755c02-d38c-4a48-858f-c610b040a91a None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:32.655884 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 324/651] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:32.657392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc755c02-d38c-4a48-858f-c610b040a91a None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:32.770835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2adb2ec-2752-446c-bf36-e21e9a2578f0 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:32.810178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd48399-f7de-4368-a75c-60131b615933 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:32.810823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd48399-f7de-4368-a75c-60131b615933 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:32.811093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd48399-f7de-4368-a75c-60131b615933 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:32.811367 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd48399-f7de-4368-a75c-60131b615933 None admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:32.812066 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc755c02-d38c-4a48-858f-c610b040a91a None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:32.812775 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd48399-f7de-4368-a75c-60131b615933 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:32.812775 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd48399-f7de-4368-a75c-60131b615933 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:32.813796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd48399-f7de-4368-a75c-60131b615933 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:32.827258 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 328/652] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:43:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:32.835882 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd48399-f7de-4368-a75c-60131b615933 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:32.867032 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4281dd-51f0-4cf9-a9ba-86084aef91cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:32.867669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4281dd-51f0-4cf9-a9ba-86084aef91cd None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:32.867951 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4281dd-51f0-4cf9-a9ba-86084aef91cd None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:32.868220 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4281dd-51f0-4cf9-a9ba-86084aef91cd None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:32.878055 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4281dd-51f0-4cf9-a9ba-86084aef91cd None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:32.879554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4281dd-51f0-4cf9-a9ba-86084aef91cd None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:32.882490 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:32.893146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-9f4281dd-51f0-4cf9-a9ba-86084aef91cd None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 18 00:43:32.893146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:32.893146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:32.893146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:32.893146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:32.893146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:32.893146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:32.893146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:32.893146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:32.893146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:32.893146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:32.893146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:32.893146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:43:32.893146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:43:32.893146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:43:32.893146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:43:32.894162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:32.894162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:32.894162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:43:32.894162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:32.894162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:43:32.894162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:32.894162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:32.894162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:32.894162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:32.894162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:32.894162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:32.894162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:32.894162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:32.894162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:32.894162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:32.894162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:32.894162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:32.894162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:32.895373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:32.895373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:32.895373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:43:32.895373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:43:32.895373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:43:32.895373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:43:32.895373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:43:32.895373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:43:32.895373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 18 00:43:32.895373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:32.895373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 325/653] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:43:32 2022] GET /identity/v3/users/neutron => generated 85 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:32.895373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4281dd-51f0-4cf9-a9ba-86084aef91cd None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:32.920041 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:32.977254 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd48399-f7de-4368-a75c-60131b615933 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:33.022106 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4281dd-51f0-4cf9-a9ba-86084aef91cd None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:33.031020 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e978c6-2ba0-4542-a1b3-77a9333e4dfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:33.032016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e978c6-2ba0-4542-a1b3-77a9333e4dfd None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:33.032286 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e978c6-2ba0-4542-a1b3-77a9333e4dfd None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:33.032620 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e978c6-2ba0-4542-a1b3-77a9333e4dfd None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:33.033859 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e978c6-2ba0-4542-a1b3-77a9333e4dfd None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:33.034292 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e978c6-2ba0-4542-a1b3-77a9333e4dfd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:33.035966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e978c6-2ba0-4542-a1b3-77a9333e4dfd None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:33.059821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 329/654] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Sep 18 00:43:32 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:33.060823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e978c6-2ba0-4542-a1b3-77a9333e4dfd None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:33.067183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae21cc9-fa66-4244-bc23-c7bdc2bfb615 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:33.068119 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae21cc9-fa66-4244-bc23-c7bdc2bfb615 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:33.068570 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae21cc9-fa66-4244-bc23-c7bdc2bfb615 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:33.069011 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae21cc9-fa66-4244-bc23-c7bdc2bfb615 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:33.081977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae21cc9-fa66-4244-bc23-c7bdc2bfb615 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:33.084440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae21cc9-fa66-4244-bc23-c7bdc2bfb615 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:33.105776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-dae21cc9-fa66-4244-bc23-c7bdc2bfb615 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 18 00:43:33.105776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:33.105776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:33.105776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:33.105776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:33.105776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:33.105776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:33.105776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:33.105776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:33.105776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:33.105776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:33.105776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:33.105776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:43:33.105776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:43:33.105776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:43:33.105776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:43:33.106869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:33.106869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:33.106869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:43:33.106869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:33.106869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:43:33.106869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:33.106869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:33.106869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:33.106869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:33.106869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:33.106869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:33.106869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:33.106869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:33.106869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:33.106869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:33.106869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:33.106869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:33.106869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:33.107808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:33.107808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:33.107808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:43:33.107808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:43:33.107808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:43:33.107808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:43:33.107808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:43:33.107808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:43:33.107808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 18 00:43:33.107808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:33.107808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 326/655] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:43:32 2022] GET /identity/v3/users/cinder => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:33.109225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae21cc9-fa66-4244-bc23-c7bdc2bfb615 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:33.203330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:33.234647 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e978c6-2ba0-4542-a1b3-77a9333e4dfd None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:33.249405 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:33.253502 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae21cc9-fa66-4244-bc23-c7bdc2bfb615 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:33.281302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d000226-6b98-4121-9dfc-49c46998a3b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:33.281918 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d000226-6b98-4121-9dfc-49c46998a3b4 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:33.282196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d000226-6b98-4121-9dfc-49c46998a3b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:33.282481 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d000226-6b98-4121-9dfc-49c46998a3b4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:33.283247 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d000226-6b98-4121-9dfc-49c46998a3b4 None admin] RBAC: Adding query filter params (role.id=81523e544c7e4993af4d1e06ce3754bc, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=d7fe9d21099e4c4891c28687bb673214) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:33.283884 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d000226-6b98-4121-9dfc-49c46998a3b4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:33.285036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d000226-6b98-4121-9dfc-49c46998a3b4 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:33.291927 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 330/656] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:43:33 2022] GET /identity/v3/role_assignments?user.id=d7fe9d21099e4c4891c28687bb673214&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=81523e544c7e4993af4d1e06ce3754bc => generated 608 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:33.293383 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d000226-6b98-4121-9dfc-49c46998a3b4 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:33.307971 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d29f0c-8274-4e26-92d3-3340b8251ced None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:33.308374 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d29f0c-8274-4e26-92d3-3340b8251ced None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:33.308657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d29f0c-8274-4e26-92d3-3340b8251ced None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:33.308929 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d29f0c-8274-4e26-92d3-3340b8251ced None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:33.309705 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d29f0c-8274-4e26-92d3-3340b8251ced None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:33.310017 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d29f0c-8274-4e26-92d3-3340b8251ced None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:33.311677 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d29f0c-8274-4e26-92d3-3340b8251ced None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:33.338557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 327/657] 10.222.0.42 () {56 vars in 1139 bytes} [Sun Sep 18 00:43:33 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:33.338557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d29f0c-8274-4e26-92d3-3340b8251ced None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:33.498377 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d000226-6b98-4121-9dfc-49c46998a3b4 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:33.540957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d29f0c-8274-4e26-92d3-3340b8251ced None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:33.584418 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:33.595097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d893a99-a86b-4f4f-b201-79a42b91da44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:33.596849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d893a99-a86b-4f4f-b201-79a42b91da44 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:33.597280 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d893a99-a86b-4f4f-b201-79a42b91da44 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:33.597683 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d893a99-a86b-4f4f-b201-79a42b91da44 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:33.626415 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01caecec-8c01-4baa-9911-72d4268d74a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:33.627310 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01caecec-8c01-4baa-9911-72d4268d74a7 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:33.627721 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01caecec-8c01-4baa-9911-72d4268d74a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:33.629105 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01caecec-8c01-4baa-9911-72d4268d74a7 None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:33.629414 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01caecec-8c01-4baa-9911-72d4268d74a7 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:33.629932 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01caecec-8c01-4baa-9911-72d4268d74a7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:33.633375 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01caecec-8c01-4baa-9911-72d4268d74a7 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:33.672632 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 328/658] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Sep 18 00:43:33 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 337 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:33.674675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01caecec-8c01-4baa-9911-72d4268d74a7 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:33.686448 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:33.741877 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d893a99-a86b-4f4f-b201-79a42b91da44 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:33.745187 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d893a99-a86b-4f4f-b201-79a42b91da44 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:33.824151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01caecec-8c01-4baa-9911-72d4268d74a7 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:33.878337 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7984ac-428c-4ff0-ae20-c7e401b06754 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:33.882041 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7984ac-428c-4ff0-ae20-c7e401b06754 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:33.882326 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7984ac-428c-4ff0-ae20-c7e401b06754 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:33.882596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7984ac-428c-4ff0-ae20-c7e401b06754 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:33.950116 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 331/659] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:43:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 658 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:43:33.951737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d893a99-a86b-4f4f-b201-79a42b91da44 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:33.959946 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:33.993769 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7984ac-428c-4ff0-ae20-c7e401b06754 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:33.997258 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7984ac-428c-4ff0-ae20-c7e401b06754 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:34.054762 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:34.092138 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d893a99-a86b-4f4f-b201-79a42b91da44 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:34.144650 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 329/660] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:43:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 471 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:43:34.146331 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0624a9d-da59-4bc1-8775-0b890ff9eccb None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:34.146818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0624a9d-da59-4bc1-8775-0b890ff9eccb None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:34.147280 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0624a9d-da59-4bc1-8775-0b890ff9eccb None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:34.148828 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 330/661] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:34.149803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0624a9d-da59-4bc1-8775-0b890ff9eccb None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:34.161277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d966a8e5-d837-4497-82f1-6526402772c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:34.162083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d966a8e5-d837-4497-82f1-6526402772c2 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:34.162533 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d966a8e5-d837-4497-82f1-6526402772c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:34.163230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d966a8e5-d837-4497-82f1-6526402772c2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:34.262019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d966a8e5-d837-4497-82f1-6526402772c2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:34.265049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d966a8e5-d837-4497-82f1-6526402772c2 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:34.267792 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:34.294117 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0624a9d-da59-4bc1-8775-0b890ff9eccb None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:34.359547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d0ae7b-220a-4b2d-8909-26bb1fbad385 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:34.360505 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d0ae7b-220a-4b2d-8909-26bb1fbad385 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:34.360958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d0ae7b-220a-4b2d-8909-26bb1fbad385 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:34.361360 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d0ae7b-220a-4b2d-8909-26bb1fbad385 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:34.378379 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d0ae7b-220a-4b2d-8909-26bb1fbad385 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:34.380962 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d0ae7b-220a-4b2d-8909-26bb1fbad385 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:34.383475 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:34.405804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-72d0ae7b-220a-4b2d-8909-26bb1fbad385 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:34.405804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:34.405804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:34.405804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:34.405804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:34.405804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:34.405804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:34.405804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:34.405804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:34.405804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:34.405804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:34.405804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:34.405804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:34.405804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:34.405804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:34.405804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:34.406833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:34.406833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:34.406833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:34.406833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:34.406833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:34.406833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:34.406833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:34.406833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:34.406833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:34.406833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:34.406833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:34.406833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:34.406833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:34.406833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:34.406833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:34.406833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:34.406833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:34.407721 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:34.407721 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:34.407721 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:34.407721 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 331/662] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:34.409060 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d0ae7b-220a-4b2d-8909-26bb1fbad385 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:34.430554 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 332/663] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:43:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 480 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:43:34.433572 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0447e747-d03c-4dc4-bd89-8f8f47aee36b None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:34.434202 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0447e747-d03c-4dc4-bd89-8f8f47aee36b None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:34.434863 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0447e747-d03c-4dc4-bd89-8f8f47aee36b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:34.456541 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-0447e747-d03c-4dc4-bd89-8f8f47aee36b None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:34.458050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-0447e747-d03c-4dc4-bd89-8f8f47aee36b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:34.574499 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d0ae7b-220a-4b2d-8909-26bb1fbad385 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:34.574822 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 333/664] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:43:34.574822 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0447e747-d03c-4dc4-bd89-8f8f47aee36b None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:34.613899 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:34.624290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700b3eee-3798-46ee-ad5c-fed19e5a3342 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:34.625073 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700b3eee-3798-46ee-ad5c-fed19e5a3342 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:34.625359 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700b3eee-3798-46ee-ad5c-fed19e5a3342 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:34.625639 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700b3eee-3798-46ee-ad5c-fed19e5a3342 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:34.639339 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700b3eee-3798-46ee-ad5c-fed19e5a3342 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:34.640850 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700b3eee-3798-46ee-ad5c-fed19e5a3342 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:34.660728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-700b3eee-3798-46ee-ad5c-fed19e5a3342 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:34.660728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:34.660728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:34.660728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:34.660728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:34.660728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:34.660728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:34.660728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:34.660728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:34.660728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:34.660728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:34.660728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:34.660728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:34.660728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:34.660728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:34.660728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:34.662574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:34.662574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:34.662574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:34.662574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:34.662574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:34.662574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:34.662574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:34.662574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:34.662574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:34.662574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:34.662574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:34.662574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:34.662574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:34.662574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:34.662574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:34.662574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:34.662574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:34.666218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:34.666218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:34.666218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:34.666218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 332/665] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:34.666218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700b3eee-3798-46ee-ad5c-fed19e5a3342 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:34.747472 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0447e747-d03c-4dc4-bd89-8f8f47aee36b None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:34.750213 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:34.788849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd184658-d0e2-4e2d-ac3a-42ca4730613b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:34.789468 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd184658-d0e2-4e2d-ac3a-42ca4730613b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:34.789740 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd184658-d0e2-4e2d-ac3a-42ca4730613b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:34.790002 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd184658-d0e2-4e2d-ac3a-42ca4730613b None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:34.790683 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd184658-d0e2-4e2d-ac3a-42ca4730613b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:34.790982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd184658-d0e2-4e2d-ac3a-42ca4730613b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:34.792714 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd184658-d0e2-4e2d-ac3a-42ca4730613b None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:34.810346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 334/666] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:34.811399 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd184658-d0e2-4e2d-ac3a-42ca4730613b None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:34.832884 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700b3eee-3798-46ee-ad5c-fed19e5a3342 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:34.907709 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b106e7-af03-49e5-8c77-d6e533008bf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:34.909288 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b106e7-af03-49e5-8c77-d6e533008bf7 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:34.910169 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b106e7-af03-49e5-8c77-d6e533008bf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:34.910986 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b106e7-af03-49e5-8c77-d6e533008bf7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:34.926830 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd184658-d0e2-4e2d-ac3a-42ca4730613b None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:34.934201 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b106e7-af03-49e5-8c77-d6e533008bf7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:34.937474 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b106e7-af03-49e5-8c77-d6e533008bf7 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:34.963716 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d84aab-11f8-41f1-9b22-5f0cc462b874 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:34.964870 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d84aab-11f8-41f1-9b22-5f0cc462b874 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:34.965146 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d84aab-11f8-41f1-9b22-5f0cc462b874 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:34.965414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d84aab-11f8-41f1-9b22-5f0cc462b874 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:34.966060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d84aab-11f8-41f1-9b22-5f0cc462b874 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:34.966357 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d84aab-11f8-41f1-9b22-5f0cc462b874 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:34.967817 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d84aab-11f8-41f1-9b22-5f0cc462b874 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:34.970134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-14b106e7-af03-49e5-8c77-d6e533008bf7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:34.970134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:34.970134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:34.970134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:34.970134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:34.970134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:34.970134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:34.970134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:34.970134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:34.970134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:34.970134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:34.970134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:34.970134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:34.970134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:34.970134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:34.970134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:34.972116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:34.972116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:34.972116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:34.972116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:34.972116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:34.972116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:34.972116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:34.972116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:34.972116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:34.972116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:34.972116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:34.972116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:34.972116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:34.972116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:34.972116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:34.972116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:34.972116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:34.974020 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:34.974020 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:34.974020 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:34.976910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 333/667] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:34.978882 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b106e7-af03-49e5-8c77-d6e533008bf7 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:34.981889 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 335/668] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:34.985326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d84aab-11f8-41f1-9b22-5f0cc462b874 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:35.014818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:35.044273 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:35.100570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d84aab-11f8-41f1-9b22-5f0cc462b874 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:35.142340 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdebc0e-b722-4e74-a699-50074f9eaf61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:35.143447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdebc0e-b722-4e74-a699-50074f9eaf61 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:35.143911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdebc0e-b722-4e74-a699-50074f9eaf61 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:35.144388 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdebc0e-b722-4e74-a699-50074f9eaf61 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:35.145562 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bdebc0e-b722-4e74-a699-50074f9eaf61 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:35.146090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bdebc0e-b722-4e74-a699-50074f9eaf61 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:35.148701 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bdebc0e-b722-4e74-a699-50074f9eaf61 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:35.157153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b106e7-af03-49e5-8c77-d6e533008bf7 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:35.176324 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 336/669] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:35.176324 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdebc0e-b722-4e74-a699-50074f9eaf61 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:35.209700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1623e6-4f09-450a-91be-46fe600b3f8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:35.210734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1623e6-4f09-450a-91be-46fe600b3f8f None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:35.211179 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1623e6-4f09-450a-91be-46fe600b3f8f None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:35.211633 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1623e6-4f09-450a-91be-46fe600b3f8f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:35.225191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1623e6-4f09-450a-91be-46fe600b3f8f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:35.227464 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1623e6-4f09-450a-91be-46fe600b3f8f None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:35.247451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-4f1623e6-4f09-450a-91be-46fe600b3f8f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:35.247451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:35.247451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:35.247451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:35.247451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:35.247451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:35.247451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:35.247451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:35.247451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:35.247451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:35.247451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:35.247451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:35.247451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:43:35.247451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:43:35.247451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:43:35.249183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:43:35.249183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:35.249183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:35.249183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:35.249183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:35.249183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:35.249183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:35.249183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:35.249183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:35.249183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:35.249183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:35.249183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:35.249183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:35.249183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:35.249183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:35.249183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:35.249183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:35.259120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:43:35.259120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:43:35.259120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:43:35.259120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:43:35.259120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:43:35.259120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:43:35.259120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:35.259120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:35.259120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 334/670] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:35.259120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1623e6-4f09-450a-91be-46fe600b3f8f None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:35.336858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:35.363714 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdebc0e-b722-4e74-a699-50074f9eaf61 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:35.407691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d8cb9e-ebc7-46e0-b153-b7549dc5423f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:35.409072 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d8cb9e-ebc7-46e0-b153-b7549dc5423f None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:35.409522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d8cb9e-ebc7-46e0-b153-b7549dc5423f None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:35.409973 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d8cb9e-ebc7-46e0-b153-b7549dc5423f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:35.420039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:35.422600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d8cb9e-ebc7-46e0-b153-b7549dc5423f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:35.425220 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d8cb9e-ebc7-46e0-b153-b7549dc5423f None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:35.435519 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1623e6-4f09-450a-91be-46fe600b3f8f None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:35.444935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-47d8cb9e-ebc7-46e0-b153-b7549dc5423f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:35.444935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:35.444935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:35.444935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:35.444935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:35.444935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:35.444935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:35.444935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:35.444935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:35.444935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:35.444935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:35.444935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:35.444935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:43:35.444935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:43:35.444935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:43:35.445899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:43:35.445899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:35.445899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:35.445899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:35.445899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:35.445899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:35.445899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:35.445899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:35.445899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:35.445899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:35.445899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:35.445899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:35.445899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:35.445899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:35.445899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:35.445899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:35.445899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:35.446810 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:43:35.446810 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:43:35.446810 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:43:35.446810 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:43:35.446810 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:43:35.446810 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:43:35.446810 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:35.446810 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:35.450829 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 337/671] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:35.452038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d8cb9e-ebc7-46e0-b153-b7549dc5423f None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:35.483012 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb93c2a-32cf-4e57-af6e-fcfb19b8c41c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:35.483663 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb93c2a-32cf-4e57-af6e-fcfb19b8c41c None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:35.483940 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb93c2a-32cf-4e57-af6e-fcfb19b8c41c None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:35.484221 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb93c2a-32cf-4e57-af6e-fcfb19b8c41c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:35.497729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb93c2a-32cf-4e57-af6e-fcfb19b8c41c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:35.504618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-0bb93c2a-32cf-4e57-af6e-fcfb19b8c41c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:35.504618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:35.504618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:35.504618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:35.504618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:35.504618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:35.504618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:35.504618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:35.504618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:35.504618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:35.504618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:35.504618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:35.504618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:35.504618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:35.504618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:35.504618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:35.505603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:35.505603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:35.505603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:35.505603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:35.505603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:35.505603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:35.505603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:35.505603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:35.505603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:35.505603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:35.505603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:35.505603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:35.505603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:35.505603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:35.505603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:35.505603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:35.506417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:35.506417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:35.506417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:35.506417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:35.506417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:35.506417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:35.506417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:35.506417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:35.506417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:35.506417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:35.506417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:35.506417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:35.506417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 335/672] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:43:35 2022] GET /identity/v3/roles/service => generated 85 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:35.507172 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb93c2a-32cf-4e57-af6e-fcfb19b8c41c None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:35.594838 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d8cb9e-ebc7-46e0-b153-b7549dc5423f None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:35.638165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009a1754-791c-4eb9-a222-5514cf926843 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:35.639342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009a1754-791c-4eb9-a222-5514cf926843 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:35.639809 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009a1754-791c-4eb9-a222-5514cf926843 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:35.640200 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009a1754-791c-4eb9-a222-5514cf926843 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:35.646198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb93c2a-32cf-4e57-af6e-fcfb19b8c41c None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:35.648083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009a1754-791c-4eb9-a222-5514cf926843 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:35.649742 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009a1754-791c-4eb9-a222-5514cf926843 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:35.663858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-009a1754-791c-4eb9-a222-5514cf926843 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:35.663858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:35.663858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:35.663858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:35.663858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:35.663858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:35.663858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:35.663858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:35.663858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:35.663858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:35.663858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:35.663858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:35.663858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:43:35.663858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:43:35.663858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:43:35.665618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:43:35.665618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:35.665618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:35.665618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:35.665618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:35.665618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:35.665618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:35.665618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:35.665618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:35.665618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:35.665618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:35.665618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:35.665618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:35.665618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:35.665618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:35.665618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:35.665618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:35.667524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:43:35.667524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:43:35.667524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:43:35.667524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:43:35.667524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:43:35.667524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:43:35.667524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:35.667524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:35.667524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 338/673] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:35.667524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009a1754-791c-4eb9-a222-5514cf926843 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:35.675672 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:35.699907 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c30aab1-7c69-405d-a152-9d9adf78d297 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:35.700586 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c30aab1-7c69-405d-a152-9d9adf78d297 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:35.700878 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c30aab1-7c69-405d-a152-9d9adf78d297 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:35.701159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c30aab1-7c69-405d-a152-9d9adf78d297 None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:35.701889 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c30aab1-7c69-405d-a152-9d9adf78d297 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:35.702201 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c30aab1-7c69-405d-a152-9d9adf78d297 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:35.703859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c30aab1-7c69-405d-a152-9d9adf78d297 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:35.718954 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 336/674] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:43:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:35.720440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c30aab1-7c69-405d-a152-9d9adf78d297 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:35.742814 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:35.823982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009a1754-791c-4eb9-a222-5514cf926843 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:35.868531 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c30aab1-7c69-405d-a152-9d9adf78d297 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:35.869068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3bf353-bd31-4690-8bea-a26072534364 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:35.869665 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3bf353-bd31-4690-8bea-a26072534364 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:35.870069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3bf353-bd31-4690-8bea-a26072534364 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:35.870987 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3bf353-bd31-4690-8bea-a26072534364 None admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:35.871832 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3bf353-bd31-4690-8bea-a26072534364 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:35.872209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3bf353-bd31-4690-8bea-a26072534364 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:35.873757 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3bf353-bd31-4690-8bea-a26072534364 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:35.890864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 339/675] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:43:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:35.892276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3bf353-bd31-4690-8bea-a26072534364 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:35.927237 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a534a7c-7b9b-4319-a3da-944ec3a024dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:35.928156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a534a7c-7b9b-4319-a3da-944ec3a024dd None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:35.928583 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a534a7c-7b9b-4319-a3da-944ec3a024dd None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:35.929045 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a534a7c-7b9b-4319-a3da-944ec3a024dd None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:35.929969 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a534a7c-7b9b-4319-a3da-944ec3a024dd None admin] RBAC: Adding query filter params (name=service) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:35.930508 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a534a7c-7b9b-4319-a3da-944ec3a024dd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:35.932239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a534a7c-7b9b-4319-a3da-944ec3a024dd None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:35.943279 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 337/676] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 18 00:43:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:35.944724 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a534a7c-7b9b-4319-a3da-944ec3a024dd None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:35.988687 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:36.047874 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3bf353-bd31-4690-8bea-a26072534364 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:36.082856 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:36.112037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7b00e8-24f4-4b54-9c27-389bd8d511b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:36.112963 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7b00e8-24f4-4b54-9c27-389bd8d511b1 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:36.113258 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7b00e8-24f4-4b54-9c27-389bd8d511b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:36.113529 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7b00e8-24f4-4b54-9c27-389bd8d511b1 None admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:36.114538 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7b00e8-24f4-4b54-9c27-389bd8d511b1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:36.115510 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7b00e8-24f4-4b54-9c27-389bd8d511b1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:36.117113 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7b00e8-24f4-4b54-9c27-389bd8d511b1 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:36.130893 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a534a7c-7b9b-4319-a3da-944ec3a024dd None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:36.145091 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 340/677] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:43:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:36.145091 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7b00e8-24f4-4b54-9c27-389bd8d511b1 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:36.193117 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c67ddc-21ae-48f7-95e9-4fa5712b56f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:36.193771 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c67ddc-21ae-48f7-95e9-4fa5712b56f6 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:36.194058 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c67ddc-21ae-48f7-95e9-4fa5712b56f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:36.194336 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c67ddc-21ae-48f7-95e9-4fa5712b56f6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:36.195188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c67ddc-21ae-48f7-95e9-4fa5712b56f6 None admin] RBAC: Adding query filter params (role.id=81523e544c7e4993af4d1e06ce3754bc, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=a59d73fd45a54651930d52300704fdb0) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:36.195508 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c67ddc-21ae-48f7-95e9-4fa5712b56f6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:36.196969 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c67ddc-21ae-48f7-95e9-4fa5712b56f6 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:36.204506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 338/678] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:43:35 2022] GET /identity/v3/role_assignments?user.id=a59d73fd45a54651930d52300704fdb0&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=81523e544c7e4993af4d1e06ce3754bc => generated 608 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:36.205357 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c67ddc-21ae-48f7-95e9-4fa5712b56f6 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:36.289396 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7b00e8-24f4-4b54-9c27-389bd8d511b1 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:36.321631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:36.337539 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab671eff-8614-4253-a7f2-ddb4d27561c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:36.338775 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab671eff-8614-4253-a7f2-ddb4d27561c9 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:36.338775 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab671eff-8614-4253-a7f2-ddb4d27561c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:36.339165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab671eff-8614-4253-a7f2-ddb4d27561c9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:36.339681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab671eff-8614-4253-a7f2-ddb4d27561c9 None admin] RBAC: Adding query filter params (role.id=81523e544c7e4993af4d1e06ce3754bc, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=9b0510812b60454586a781e6399e77a4) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:36.339995 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab671eff-8614-4253-a7f2-ddb4d27561c9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:36.341537 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab671eff-8614-4253-a7f2-ddb4d27561c9 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:36.356680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 341/679] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:43:36 2022] GET /identity/v3/role_assignments?user.id=9b0510812b60454586a781e6399e77a4&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=81523e544c7e4993af4d1e06ce3754bc => generated 608 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:36.357335 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab671eff-8614-4253-a7f2-ddb4d27561c9 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:36.405832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c67ddc-21ae-48f7-95e9-4fa5712b56f6 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:36.506263 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89367b29-6655-4d52-834f-ee02f71fb301 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:36.507803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89367b29-6655-4d52-834f-ee02f71fb301 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:36.508827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89367b29-6655-4d52-834f-ee02f71fb301 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:36.509212 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89367b29-6655-4d52-834f-ee02f71fb301 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:36.527909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:36.536892 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89367b29-6655-4d52-834f-ee02f71fb301 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:36.539348 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89367b29-6655-4d52-834f-ee02f71fb301 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:36.566082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-89367b29-6655-4d52-834f-ee02f71fb301 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:36.566082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:36.566082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:36.566082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:36.566082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:36.566082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:36.566082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:36.566082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:36.566082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:36.566082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:36.566082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:36.566082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:36.566082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:36.566082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:36.566082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:36.566082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:36.568021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:36.568021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:36.568021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:36.568021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:36.568021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:36.568021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:36.568021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:36.568021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:36.568021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:36.568021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:36.568021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:36.568021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:36.568021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:36.568021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:36.568021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:36.568021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:36.568021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:36.569736 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:36.569736 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:36.569736 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:36.570087 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab671eff-8614-4253-a7f2-ddb4d27561c9 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:36.572682 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 339/680] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 367 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:36.575646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9f9665-c50a-4f3e-9ece-8531bd9b7f1d None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:36.576613 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9f9665-c50a-4f3e-9ece-8531bd9b7f1d None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:36.577412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9f9665-c50a-4f3e-9ece-8531bd9b7f1d None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:36.580385 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 340/681] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:36 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:36.582920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9f9665-c50a-4f3e-9ece-8531bd9b7f1d None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:36.635799 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4488798e-0000-4124-b85e-f692a7f16d0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:36.637003 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4488798e-0000-4124-b85e-f692a7f16d0a None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:36.637636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4488798e-0000-4124-b85e-f692a7f16d0a None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:36.638017 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4488798e-0000-4124-b85e-f692a7f16d0a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:36.639503 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4488798e-0000-4124-b85e-f692a7f16d0a None admin] RBAC: Adding query filter params (role.id=81523e544c7e4993af4d1e06ce3754bc, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=c4c3eaee0a8f4de0b79841a95b35091f) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:36.639785 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4488798e-0000-4124-b85e-f692a7f16d0a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:36.641946 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4488798e-0000-4124-b85e-f692a7f16d0a None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:36.655826 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 342/682] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:43:36 2022] GET /identity/v3/role_assignments?user.id=c4c3eaee0a8f4de0b79841a95b35091f&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=81523e544c7e4993af4d1e06ce3754bc => generated 608 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:36.658009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d659e3b3-e3cf-4298-9da5-bdb90c46ed05 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:36.658521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d659e3b3-e3cf-4298-9da5-bdb90c46ed05 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:36.659003 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d659e3b3-e3cf-4298-9da5-bdb90c46ed05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:36.696692 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-d659e3b3-e3cf-4298-9da5-bdb90c46ed05 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:36.697962 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d659e3b3-e3cf-4298-9da5-bdb90c46ed05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:36.792700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9f9665-c50a-4f3e-9ece-8531bd9b7f1d None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:36.812075 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:36.861089 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fc75c1-cca8-4a21-b451-ae5fd1d536f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:36.862125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fc75c1-cca8-4a21-b451-ae5fd1d536f6 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:36.862676 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fc75c1-cca8-4a21-b451-ae5fd1d536f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:36.863154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fc75c1-cca8-4a21-b451-ae5fd1d536f6 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:36.864285 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97fc75c1-cca8-4a21-b451-ae5fd1d536f6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:36.864809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97fc75c1-cca8-4a21-b451-ae5fd1d536f6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:36.867400 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97fc75c1-cca8-4a21-b451-ae5fd1d536f6 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:36.870341 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 343/683] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:43:36.891119 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 341/684] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:36.899067 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d659e3b3-e3cf-4298-9da5-bdb90c46ed05 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:37.103741 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d659e3b3-e3cf-4298-9da5-bdb90c46ed05 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:37.154076 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a194d40b-2c94-4a67-a614-df34e3109d1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:37.155101 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a194d40b-2c94-4a67-a614-df34e3109d1f None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:37.155573 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a194d40b-2c94-4a67-a614-df34e3109d1f None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:37.156047 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a194d40b-2c94-4a67-a614-df34e3109d1f None admin] PATH_INFO: `/v3/users/swift` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:37.177260 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a194d40b-2c94-4a67-a614-df34e3109d1f None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:37.180095 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a194d40b-2c94-4a67-a614-df34e3109d1f None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:37.206474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-a194d40b-2c94-4a67-a614-df34e3109d1f None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 18 00:43:37.206474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:37.206474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:37.206474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:37.206474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:37.206474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:37.206474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:37.206474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:37.206474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:37.206474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:37.206474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:37.206474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:37.206474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:43:37.206474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:43:37.206474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:43:37.206474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:43:37.208278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:37.208278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:37.208278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:43:37.208278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:37.208278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:43:37.208278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:37.208278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:37.208278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:37.208278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:37.208278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:37.208278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:37.208278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:37.208278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:37.208278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:37.208278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:37.208278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:37.208278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:37.208278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:37.210134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:37.210134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:37.210134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:43:37.210134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:43:37.210134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:43:37.210134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:43:37.210134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:43:37.210134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:43:37.210134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 18 00:43:37.210134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:37.215353 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 344/685] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:43:36 2022] GET /identity/v3/users/swift => generated 83 bytes in 317 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:37.236195 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fc75c1-cca8-4a21-b451-ae5fd1d536f6 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:37.286830 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a194d40b-2c94-4a67-a614-df34e3109d1f None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:37.340699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:37.361002 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:37.442780 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fc75c1-cca8-4a21-b451-ae5fd1d536f6 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:37.486501 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ff2c36-2065-4ce0-b3ce-96dd2f70eb4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:37.487528 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ff2c36-2065-4ce0-b3ce-96dd2f70eb4d None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:37.488013 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ff2c36-2065-4ce0-b3ce-96dd2f70eb4d None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:37.488498 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ff2c36-2065-4ce0-b3ce-96dd2f70eb4d None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:37.489523 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ff2c36-2065-4ce0-b3ce-96dd2f70eb4d None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:37.490073 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ff2c36-2065-4ce0-b3ce-96dd2f70eb4d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:37.492276 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ff2c36-2065-4ce0-b3ce-96dd2f70eb4d None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:37.521517 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a194d40b-2c94-4a67-a614-df34e3109d1f None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:37.537036 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 342/686] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Sep 18 00:43:37 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:37.556905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ff2c36-2065-4ce0-b3ce-96dd2f70eb4d None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:37.600933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61eb6d76-7fba-444f-beae-fd6bcaa8cc93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:37.601609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eb6d76-7fba-444f-beae-fd6bcaa8cc93 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:37.601886 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eb6d76-7fba-444f-beae-fd6bcaa8cc93 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:37.602159 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eb6d76-7fba-444f-beae-fd6bcaa8cc93 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:37.627096 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61eb6d76-7fba-444f-beae-fd6bcaa8cc93 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:37.628633 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61eb6d76-7fba-444f-beae-fd6bcaa8cc93 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:37.664088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-61eb6d76-7fba-444f-beae-fd6bcaa8cc93 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:37.664088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:37.664088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:37.664088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:37.664088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:37.664088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:37.664088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:37.664088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:37.664088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:37.664088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:37.664088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:37.664088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:37.664088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:37.664088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:37.664088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:37.664088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:37.665117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:37.665117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:37.665117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:37.665117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:37.665117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:37.665117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:37.665117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:37.665117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:37.665117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:37.665117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:37.665117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:37.665117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:37.665117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:37.665117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:37.665117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:37.665117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:37.665117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:37.666010 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:37.666010 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:37.666010 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:37.666737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 345/687] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 380 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:43:37.678173 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61eb6d76-7fba-444f-beae-fd6bcaa8cc93 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:37.764495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ff2c36-2065-4ce0-b3ce-96dd2f70eb4d None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:37.775189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:37.814247 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09de692-9d28-4aed-88e7-b01576404e2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:37.815209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09de692-9d28-4aed-88e7-b01576404e2b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:37.815657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09de692-9d28-4aed-88e7-b01576404e2b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:37.816080 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09de692-9d28-4aed-88e7-b01576404e2b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:37.954877 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:37.969277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61eb6d76-7fba-444f-beae-fd6bcaa8cc93 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:38.037076 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e857395-7eef-4b11-ac02-3dc7863646f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:38.037615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e857395-7eef-4b11-ac02-3dc7863646f1 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:38.037889 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e857395-7eef-4b11-ac02-3dc7863646f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:38.038158 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e857395-7eef-4b11-ac02-3dc7863646f1 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:38.038891 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e857395-7eef-4b11-ac02-3dc7863646f1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:38.039194 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e857395-7eef-4b11-ac02-3dc7863646f1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:38.040837 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e857395-7eef-4b11-ac02-3dc7863646f1 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:38.045025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09de692-9d28-4aed-88e7-b01576404e2b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:38.046523 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09de692-9d28-4aed-88e7-b01576404e2b None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:38.070542 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 346/688] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 393 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:38.078850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e857395-7eef-4b11-ac02-3dc7863646f1 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:38.240990 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e857395-7eef-4b11-ac02-3dc7863646f1 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:38.270850 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 343/689] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:43:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 710 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:43:38.275349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09de692-9d28-4aed-88e7-b01576404e2b None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:38.297864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:38.345077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a806fd-4aa5-4fe9-93f0-30c0b27b8c90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:38.346037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a806fd-4aa5-4fe9-93f0-30c0b27b8c90 None admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:38.346489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a806fd-4aa5-4fe9-93f0-30c0b27b8c90 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:38.346954 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a806fd-4aa5-4fe9-93f0-30c0b27b8c90 None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:38.348012 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a806fd-4aa5-4fe9-93f0-30c0b27b8c90 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:38.350534 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a806fd-4aa5-4fe9-93f0-30c0b27b8c90 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:38.388569 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 347/690] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 18 00:43:38 2022] POST /identity/v3/users => generated 254 bytes in 310 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:43:38.486016 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09de692-9d28-4aed-88e7-b01576404e2b None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:38.551198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9435859f-687c-40f4-b62b-1651dcb85da8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:38.552188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9435859f-687c-40f4-b62b-1651dcb85da8 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:38.552618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9435859f-687c-40f4-b62b-1651dcb85da8 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:38.553034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9435859f-687c-40f4-b62b-1651dcb85da8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:38.573921 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9435859f-687c-40f4-b62b-1651dcb85da8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:38.576490 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9435859f-687c-40f4-b62b-1651dcb85da8 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:38.608019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-9435859f-687c-40f4-b62b-1651dcb85da8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:38.608019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:38.608019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:38.608019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:38.608019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:38.608019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:38.608019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:38.608019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:38.608019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:38.608019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:38.608019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:38.608019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:38.608019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:38.608019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:38.608019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:38.608019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:38.610323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:38.610323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:38.610323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:38.610323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:38.610323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:38.610323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:38.610323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:38.610323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:38.610323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:38.610323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:38.610323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:38.610323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:38.610323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:38.610323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:38.610323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:38.610323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:38.610323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:38.612565 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:38.612565 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:38.612565 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:38.612565 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 344/691] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 336 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:38.619630 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a806fd-4aa5-4fe9-93f0-30c0b27b8c90 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:38.647112 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:38.814770 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a806fd-4aa5-4fe9-93f0-30c0b27b8c90 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:38.855449 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92173b6d-c1a9-477a-a188-6bfc2fcff339 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:38.856068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92173b6d-c1a9-477a-a188-6bfc2fcff339 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:38.856346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92173b6d-c1a9-477a-a188-6bfc2fcff339 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:38.856622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92173b6d-c1a9-477a-a188-6bfc2fcff339 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:38.857279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92173b6d-c1a9-477a-a188-6bfc2fcff339 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:38.857576 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92173b6d-c1a9-477a-a188-6bfc2fcff339 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:38.859136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92173b6d-c1a9-477a-a188-6bfc2fcff339 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:38.876646 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 348/692] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:43:38.884772 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9435859f-687c-40f4-b62b-1651dcb85da8 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:39.003155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:39.081405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9435859f-687c-40f4-b62b-1651dcb85da8 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:39.160018 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b85757-87d4-4e91-8eeb-b5c11217bf16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:39.161112 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b85757-87d4-4e91-8eeb-b5c11217bf16 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:39.161598 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b85757-87d4-4e91-8eeb-b5c11217bf16 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:39.162163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b85757-87d4-4e91-8eeb-b5c11217bf16 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:39.179403 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b85757-87d4-4e91-8eeb-b5c11217bf16 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:39.182190 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b85757-87d4-4e91-8eeb-b5c11217bf16 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:39.212933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-54b85757-87d4-4e91-8eeb-b5c11217bf16 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:39.212933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:39.212933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:39.212933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:39.212933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:39.212933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:39.212933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:39.212933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:39.212933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:39.212933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:39.212933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:39.212933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:39.212933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:43:39.212933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:43:39.212933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:43:39.215229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:43:39.215229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:39.215229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:39.215229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:39.215229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:39.215229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:39.215229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:39.215229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:39.215229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:39.215229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:39.215229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:39.215229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:39.215229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:39.215229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:39.215229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:39.215229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:39.215229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:39.217370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:43:39.217370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:43:39.217370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:43:39.217370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:43:39.217370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:43:39.217370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:43:39.217370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:39.217370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:39.222710 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 345/693] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 331 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:43:39.224239 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92173b6d-c1a9-477a-a188-6bfc2fcff339 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:39.370030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:39.427953 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92173b6d-c1a9-477a-a188-6bfc2fcff339 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:39.489686 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b69521-414d-406b-8c55-452c1e7b9221 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:39.490734 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b69521-414d-406b-8c55-452c1e7b9221 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:39.491213 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b69521-414d-406b-8c55-452c1e7b9221 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:39.491686 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b69521-414d-406b-8c55-452c1e7b9221 None admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:39.492946 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b69521-414d-406b-8c55-452c1e7b9221 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:39.493479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b69521-414d-406b-8c55-452c1e7b9221 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:39.496650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b69521-414d-406b-8c55-452c1e7b9221 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:39.520147 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 349/694] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:43:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:39.540424 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b85757-87d4-4e91-8eeb-b5c11217bf16 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:39.793863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b85757-87d4-4e91-8eeb-b5c11217bf16 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:39.858786 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:39.893053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a86999-cf65-4a52-8198-31c7155677aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:39.894092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a86999-cf65-4a52-8198-31c7155677aa None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:39.894615 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a86999-cf65-4a52-8198-31c7155677aa None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:39.895083 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a86999-cf65-4a52-8198-31c7155677aa None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:39.896466 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a86999-cf65-4a52-8198-31c7155677aa None admin] RBAC: Adding query filter params (role.id=81523e544c7e4993af4d1e06ce3754bc, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=6230ed7ce0eb43e9b2b8530a28eb6299) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:39.897026 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a86999-cf65-4a52-8198-31c7155677aa None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:39.899775 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a86999-cf65-4a52-8198-31c7155677aa None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:39.921617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 346/695] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:43:39 2022] GET /identity/v3/role_assignments?user.id=6230ed7ce0eb43e9b2b8530a28eb6299&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=81523e544c7e4993af4d1e06ce3754bc => generated 608 bytes in 382 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:40.012267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776241ec-7775-4dfb-abff-7da610d9cdbe None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:40.012763 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776241ec-7775-4dfb-abff-7da610d9cdbe None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:40.013248 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776241ec-7775-4dfb-abff-7da610d9cdbe None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:40.015647 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 350/696] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:40 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:40.024548 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedcfb75-4d15-4f3e-a1e9-4fe4386b40c3 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:40.024989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedcfb75-4d15-4f3e-a1e9-4fe4386b40c3 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:40.025406 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedcfb75-4d15-4f3e-a1e9-4fe4386b40c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:40.075009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-bedcfb75-4d15-4f3e-a1e9-4fe4386b40c3 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:40.076790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-bedcfb75-4d15-4f3e-a1e9-4fe4386b40c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:40.092781 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb7d944-5dc6-4183-b8be-aa4036105169 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:40.093233 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb7d944-5dc6-4183-b8be-aa4036105169 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:40.093677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb7d944-5dc6-4183-b8be-aa4036105169 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:40.100770 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 351/697] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:40 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:40.109173 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c851125d-fba0-4b2b-8d9c-cb5827619f62 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:40.109641 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c851125d-fba0-4b2b-8d9c-cb5827619f62 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:40.110090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c851125d-fba0-4b2b-8d9c-cb5827619f62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:40.159694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-c851125d-fba0-4b2b-8d9c-cb5827619f62 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:40.161380 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c851125d-fba0-4b2b-8d9c-cb5827619f62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:40.238373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 347/698] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:43:40.281049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 352/699] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 18 00:43:40.553454 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedcfb75-4d15-4f3e-a1e9-4fe4386b40c3 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:40.678497 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedcfb75-4d15-4f3e-a1e9-4fe4386b40c3 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:40.704863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:40.761744 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd1a12d-4afc-48d0-abdd-34a134b1f501 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:40.763696 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd1a12d-4afc-48d0-abdd-34a134b1f501 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:40.764108 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd1a12d-4afc-48d0-abdd-34a134b1f501 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:40.764501 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd1a12d-4afc-48d0-abdd-34a134b1f501 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:40.765454 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c851125d-fba0-4b2b-8d9c-cb5827619f62 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:40.800066 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd1a12d-4afc-48d0-abdd-34a134b1f501 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:40.809972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-ddd1a12d-4afc-48d0-abdd-34a134b1f501 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:43:40.809972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:40.809972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:40.809972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:40.809972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:40.809972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:40.809972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:40.809972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:40.809972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:40.809972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:40.809972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:40.809972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:40.809972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:40.809972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:40.809972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:40.809972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:40.811070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:40.811070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:40.811070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:40.811070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:40.811070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:40.811070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:40.811070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:40.811070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:40.811070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:40.811070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:40.811070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:40.811070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:40.811070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:40.811070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:40.811070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:40.811070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:40.811910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:40.811910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:40.811910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:40.811910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:40.811910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:40.811910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:40.811910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:40.811910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:40.811910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:40.811910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:40.811910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:43:40.811910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:40.820781 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 348/700] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:43:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:40.833584 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd1a12d-4afc-48d0-abdd-34a134b1f501 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:40.851690 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:40.903420 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c851125d-fba0-4b2b-8d9c-cb5827619f62 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:40.943542 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba851fc-37d3-45c6-9f1f-a29bf456c906 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:40.944613 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba851fc-37d3-45c6-9f1f-a29bf456c906 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:40.945101 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba851fc-37d3-45c6-9f1f-a29bf456c906 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:40.945591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba851fc-37d3-45c6-9f1f-a29bf456c906 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:40.946400 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba851fc-37d3-45c6-9f1f-a29bf456c906 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:40.948509 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba851fc-37d3-45c6-9f1f-a29bf456c906 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:40.962149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-dba851fc-37d3-45c6-9f1f-a29bf456c906 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 18 00:43:40.962149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:40.962149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:43:40.962149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:43:40.962149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:43:40.962149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:43:40.962149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:43:40.962149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:43:40.962149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 18 00:43:40.962149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:40.962149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:43:40.962149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:40.962149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:40.962149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:40.962149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:40.962149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:40.962149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:40.963237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:40.963237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:40.963237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:40.963237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:40.963237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:40.963237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:40.963237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:43:40.963237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:43:40.963237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:43:40.963237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:43:40.963237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:40.963237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:40.963237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:40.963237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:40.963237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:40.963237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:40.964657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:40.964657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:40.964657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:40.964657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:40.964657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:40.964657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:40.964657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:40.964657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:40.964657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:40.964657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:40.964657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:43:40.964657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:43:40.964657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 18 00:43:40.964657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:40.967548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 353/701] 10.222.0.42 () {56 vars in 1129 bytes} [Sun Sep 18 00:43:40 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:40.973414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d814f5a-2366-4fba-9cd7-6e13724fc5bd None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:40.973911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d814f5a-2366-4fba-9cd7-6e13724fc5bd None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:40.974388 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d814f5a-2366-4fba-9cd7-6e13724fc5bd None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:40.976954 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 354/702] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:40 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:40.977804 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d814f5a-2366-4fba-9cd7-6e13724fc5bd None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:41.065973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd1a12d-4afc-48d0-abdd-34a134b1f501 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:41.104742 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cbfe05-b733-4ee3-8657-72dcdb4266c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:41.105780 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cbfe05-b733-4ee3-8657-72dcdb4266c0 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:41.106274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cbfe05-b733-4ee3-8657-72dcdb4266c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:41.106804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cbfe05-b733-4ee3-8657-72dcdb4266c0 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:41.107700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cbfe05-b733-4ee3-8657-72dcdb4266c0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:41.108215 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cbfe05-b733-4ee3-8657-72dcdb4266c0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:41.109864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cbfe05-b733-4ee3-8657-72dcdb4266c0 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:41.124936 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 349/703] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:43:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:43:41.126683 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe01346a-c6ed-40f6-8f03-273d39abe00b None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:41.127194 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe01346a-c6ed-40f6-8f03-273d39abe00b None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:41.127765 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe01346a-c6ed-40f6-8f03-273d39abe00b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:41.147422 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-fe01346a-c6ed-40f6-8f03-273d39abe00b None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:41.148992 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-fe01346a-c6ed-40f6-8f03-273d39abe00b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:41.170316 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d814f5a-2366-4fba-9cd7-6e13724fc5bd None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:41.182396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:41.217291 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:41.234890 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 350/704] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:43:41.236009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea4ce6f-b71b-43f0-885c-33f3c077c268 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:41.236532 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe01346a-c6ed-40f6-8f03-273d39abe00b None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:41.236877 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea4ce6f-b71b-43f0-885c-33f3c077c268 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:41.237338 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea4ce6f-b71b-43f0-885c-33f3c077c268 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:41.237735 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea4ce6f-b71b-43f0-885c-33f3c077c268 None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:41.238930 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea4ce6f-b71b-43f0-885c-33f3c077c268 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:41.239477 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea4ce6f-b71b-43f0-885c-33f3c077c268 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:41.241989 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea4ce6f-b71b-43f0-885c-33f3c077c268 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:41.250196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 355/705] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Sep 18 00:43:40 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:41.257174 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea4ce6f-b71b-43f0-885c-33f3c077c268 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:41.479435 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe01346a-c6ed-40f6-8f03-273d39abe00b None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:41.526143 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf5a674-a7aa-459f-a23a-e3a276080bc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:41.527183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf5a674-a7aa-459f-a23a-e3a276080bc0 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:41.527633 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf5a674-a7aa-459f-a23a-e3a276080bc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:41.528063 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf5a674-a7aa-459f-a23a-e3a276080bc0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:41.537341 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea4ce6f-b71b-43f0-885c-33f3c077c268 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:41.542738 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf5a674-a7aa-459f-a23a-e3a276080bc0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:41.544665 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf5a674-a7aa-459f-a23a-e3a276080bc0 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:41.563482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-aaf5a674-a7aa-459f-a23a-e3a276080bc0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:41.563482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:41.563482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:41.563482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:41.563482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:41.563482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:41.563482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:41.563482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:41.563482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:41.563482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:41.563482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:41.563482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:41.563482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:41.563482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:41.563482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:41.563482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:41.565646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:41.565646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:41.565646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:41.565646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:41.565646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:41.565646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:41.565646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:41.565646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:41.565646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:41.565646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:41.565646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:41.565646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:41.565646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:41.565646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:41.565646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:41.565646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:41.565646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:41.567610 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:41.567610 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:41.567610 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:41.571554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 351/706] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 331 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:41.573110 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614b7140-9e82-4cb1-973a-348e82cac9c2 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:41.573951 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614b7140-9e82-4cb1-973a-348e82cac9c2 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:41.574236 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614b7140-9e82-4cb1-973a-348e82cac9c2 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:41.575681 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 352/707] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:41.576502 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614b7140-9e82-4cb1-973a-348e82cac9c2 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:41.603791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:41.606170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964cbcc8-0996-4d62-b15e-efe59892666a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:41.607113 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964cbcc8-0996-4d62-b15e-efe59892666a None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:41.607564 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964cbcc8-0996-4d62-b15e-efe59892666a None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:41.607975 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964cbcc8-0996-4d62-b15e-efe59892666a None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:41.608802 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964cbcc8-0996-4d62-b15e-efe59892666a None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:41.609112 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964cbcc8-0996-4d62-b15e-efe59892666a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:41.610571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964cbcc8-0996-4d62-b15e-efe59892666a None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:41.626505 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 356/708] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Sep 18 00:43:41 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 362 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:43:41.626505 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964cbcc8-0996-4d62-b15e-efe59892666a None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:41.720669 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:41.728128 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614b7140-9e82-4cb1-973a-348e82cac9c2 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:41.773934 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b159fec-3cb4-4d2f-b957-7c91b77415af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:41.774492 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b159fec-3cb4-4d2f-b957-7c91b77415af None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:41.774777 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b159fec-3cb4-4d2f-b957-7c91b77415af None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:41.775049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b159fec-3cb4-4d2f-b957-7c91b77415af None admin] PATH_INFO: `/v3/services/network` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:41.775632 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b159fec-3cb4-4d2f-b957-7c91b77415af None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:41.777610 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b159fec-3cb4-4d2f-b957-7c91b77415af None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:41.790223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-2b159fec-3cb4-4d2f-b957-7c91b77415af None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 18 00:43:41.790223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:41.790223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:43:41.790223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:43:41.790223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:43:41.790223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:43:41.790223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:43:41.790223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:43:41.790223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 18 00:43:41.790223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:41.790223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:43:41.790223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:41.790223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:41.790223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:41.790223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:41.790223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:41.790223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:41.791647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:41.791647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:41.791647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:41.791647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:41.791647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:41.791647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:41.791647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:43:41.791647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:43:41.791647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:43:41.791647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:43:41.791647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:41.791647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:41.791647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:41.791647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:41.791647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:41.791647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:41.793021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:41.793021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:41.793021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:41.793021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:41.793021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:41.793021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:41.793021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:41.793021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:41.793021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:41.793021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:41.793021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:43:41.793021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:43:41.793021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 18 00:43:41.793021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:41.802093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964cbcc8-0996-4d62-b15e-efe59892666a None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:41.803086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 353/709] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:41 2022] GET /identity/v3/services/network => generated 88 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:41.803086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c9696a-4f79-4c46-a752-c35cfe2eac67 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:41.803086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c9696a-4f79-4c46-a752-c35cfe2eac67 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:41.803086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c9696a-4f79-4c46-a752-c35cfe2eac67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:41.819843 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-c7c9696a-4f79-4c46-a752-c35cfe2eac67 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:41.821084 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c7c9696a-4f79-4c46-a752-c35cfe2eac67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:41.846638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0798d2-53c4-40b6-8549-4890ad8abd74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:41.847313 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0798d2-53c4-40b6-8549-4890ad8abd74 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:41.847588 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0798d2-53c4-40b6-8549-4890ad8abd74 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:41.847878 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0798d2-53c4-40b6-8549-4890ad8abd74 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:41.848551 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0798d2-53c4-40b6-8549-4890ad8abd74 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:41.848853 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0798d2-53c4-40b6-8549-4890ad8abd74 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:41.850278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0798d2-53c4-40b6-8549-4890ad8abd74 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:41.864928 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 357/710] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:41.865977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0798d2-53c4-40b6-8549-4890ad8abd74 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:41.918017 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 354/711] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:43:41.918526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c9696a-4f79-4c46-a752-c35cfe2eac67 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:41.936018 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:42.022218 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0798d2-53c4-40b6-8549-4890ad8abd74 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:42.041457 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:42.052569 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c9696a-4f79-4c46-a752-c35cfe2eac67 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:42.078819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8da80c2-7d0b-4f16-8f4a-ee6894d395ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:42.081845 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8da80c2-7d0b-4f16-8f4a-ee6894d395ec None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:42.081845 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8da80c2-7d0b-4f16-8f4a-ee6894d395ec None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:42.081845 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8da80c2-7d0b-4f16-8f4a-ee6894d395ec None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:42.081845 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8da80c2-7d0b-4f16-8f4a-ee6894d395ec None admin] RBAC: Adding query filter params (name=network) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:42.082965 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8da80c2-7d0b-4f16-8f4a-ee6894d395ec None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:42.085192 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8da80c2-7d0b-4f16-8f4a-ee6894d395ec None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:42.098713 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 358/712] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 18 00:43:41 2022] GET /identity/v3/services?name=network => generated 124 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:42.105199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8da80c2-7d0b-4f16-8f4a-ee6894d395ec None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:42.121250 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0a52a6-1d69-4355-921d-c98f003a80bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:42.121915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0a52a6-1d69-4355-921d-c98f003a80bf None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:42.122190 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0a52a6-1d69-4355-921d-c98f003a80bf None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:42.123597 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0a52a6-1d69-4355-921d-c98f003a80bf None admin] PATH_INFO: `/v3/users/nova` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:42.134661 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a0a52a6-1d69-4355-921d-c98f003a80bf None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:42.136922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a0a52a6-1d69-4355-921d-c98f003a80bf None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:42.150876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-3a0a52a6-1d69-4355-921d-c98f003a80bf None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 18 00:43:42.150876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:42.150876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:42.150876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:42.150876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:42.150876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:42.150876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:42.150876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:42.150876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:42.150876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:42.150876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:42.150876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:42.150876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:43:42.150876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:43:42.150876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:43:42.150876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:43:42.153974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:42.153974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:42.153974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:43:42.153974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:42.153974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:43:42.153974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:42.153974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:42.153974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:42.153974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:42.153974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:42.153974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:42.153974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:42.153974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:42.153974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:42.153974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:42.153974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:42.153974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:42.153974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:42.157094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:42.157094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:42.157094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:43:42.157094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:43:42.157094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:43:42.157094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:43:42.157094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:43:42.157094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:43:42.157094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 18 00:43:42.157094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:42.157094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 355/713] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Sep 18 00:43:41 2022] GET /identity/v3/users/nova => generated 82 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:42.163462 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0a52a6-1d69-4355-921d-c98f003a80bf None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:42.252894 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8da80c2-7d0b-4f16-8f4a-ee6894d395ec None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:42.320167 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:42.341977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0a52a6-1d69-4355-921d-c98f003a80bf None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:42.359345 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1c7d06-7b80-4332-92be-fb48d3eeda27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:42.359980 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1c7d06-7b80-4332-92be-fb48d3eeda27 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:42.360248 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1c7d06-7b80-4332-92be-fb48d3eeda27 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:42.360522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1c7d06-7b80-4332-92be-fb48d3eeda27 None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:42.361177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1c7d06-7b80-4332-92be-fb48d3eeda27 None admin] RBAC: Adding query filter params (type=network) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:42.361482 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1c7d06-7b80-4332-92be-fb48d3eeda27 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:42.365024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1c7d06-7b80-4332-92be-fb48d3eeda27 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:42.382523 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 359/714] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 18 00:43:42 2022] GET /identity/v3/services?type=network => generated 124 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:42.393323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c407ff-f2d7-4a86-9bbb-5084e9427479 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:42.394029 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c407ff-f2d7-4a86-9bbb-5084e9427479 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:42.394276 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c407ff-f2d7-4a86-9bbb-5084e9427479 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:42.394555 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c407ff-f2d7-4a86-9bbb-5084e9427479 None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:42.395282 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c407ff-f2d7-4a86-9bbb-5084e9427479 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:42.395651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c407ff-f2d7-4a86-9bbb-5084e9427479 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:42.397055 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c407ff-f2d7-4a86-9bbb-5084e9427479 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:42.427971 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 356/715] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Sep 18 00:43:42 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:42.435595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1c7d06-7b80-4332-92be-fb48d3eeda27 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:42.486103 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c407ff-f2d7-4a86-9bbb-5084e9427479 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:42.513062 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:42.596708 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1c7d06-7b80-4332-92be-fb48d3eeda27 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:42.641789 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8116ab1-2ab7-4c77-9ede-4b4dfa0194f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:42.642727 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8116ab1-2ab7-4c77-9ede-4b4dfa0194f1 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:42.643005 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8116ab1-2ab7-4c77-9ede-4b4dfa0194f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:42.643289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8116ab1-2ab7-4c77-9ede-4b4dfa0194f1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:42.733680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8116ab1-2ab7-4c77-9ede-4b4dfa0194f1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:42.735249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8116ab1-2ab7-4c77-9ede-4b4dfa0194f1 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:42.744341 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c407ff-f2d7-4a86-9bbb-5084e9427479 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:42.790894 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:42.807164 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:42.820599 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39748c91-b6c5-4fa9-8be1-4c124cfc2fa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:42.822671 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39748c91-b6c5-4fa9-8be1-4c124cfc2fa0 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:42.823984 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39748c91-b6c5-4fa9-8be1-4c124cfc2fa0 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:42.824738 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39748c91-b6c5-4fa9-8be1-4c124cfc2fa0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:42.866324 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39748c91-b6c5-4fa9-8be1-4c124cfc2fa0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:42.878827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-39748c91-b6c5-4fa9-8be1-4c124cfc2fa0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:42.878827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:42.878827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:42.878827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:42.878827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:42.878827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:42.878827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:42.878827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:42.878827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:42.878827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:42.878827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:42.878827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:42.878827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:42.878827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:42.878827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:42.878827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:42.880638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:42.880638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:42.880638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:42.880638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:42.880638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:42.880638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:42.880638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:42.880638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:42.880638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:42.880638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:42.880638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:42.880638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:42.880638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:42.880638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:42.880638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:42.880638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:42.882240 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:42.882240 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:42.882240 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:42.882240 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:42.882240 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:42.882240 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:42.882240 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:42.882240 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:42.882240 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:42.882240 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:42.882240 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:42.882240 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:42.888017 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 357/716] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:43:42 2022] GET /identity/v3/roles/service => generated 85 bytes in 403 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:42.898508 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39748c91-b6c5-4fa9-8be1-4c124cfc2fa0 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:42.934623 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 360/717] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:43:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 501 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:43:42.940665 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8116ab1-2ab7-4c77-9ede-4b4dfa0194f1 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:43.121712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39748c91-b6c5-4fa9-8be1-4c124cfc2fa0 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:43.158214 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8116ab1-2ab7-4c77-9ede-4b4dfa0194f1 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:43.228796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d603abd7-bacd-4ebd-8234-ca882b4ae7a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:43.230423 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d603abd7-bacd-4ebd-8234-ca882b4ae7a3 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:43.231229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d603abd7-bacd-4ebd-8234-ca882b4ae7a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:43.232497 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d603abd7-bacd-4ebd-8234-ca882b4ae7a3 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:43.234344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d603abd7-bacd-4ebd-8234-ca882b4ae7a3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:43.236315 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d603abd7-bacd-4ebd-8234-ca882b4ae7a3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:43.240546 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d603abd7-bacd-4ebd-8234-ca882b4ae7a3 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:43.251932 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9603aa4-157d-4c2d-aa4f-08b8144db2a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:43.258391 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9603aa4-157d-4c2d-aa4f-08b8144db2a8 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:43.259371 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9603aa4-157d-4c2d-aa4f-08b8144db2a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:43.261360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9603aa4-157d-4c2d-aa4f-08b8144db2a8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:43.265086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 358/718] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 18 00:43:42 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 368 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:43.275633 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d603abd7-bacd-4ebd-8234-ca882b4ae7a3 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:43.289477 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:43.295439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9603aa4-157d-4c2d-aa4f-08b8144db2a8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:43.299953 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9603aa4-157d-4c2d-aa4f-08b8144db2a8 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:43.339958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-e9603aa4-157d-4c2d-aa4f-08b8144db2a8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:43.339958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:43.339958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:43.339958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:43.339958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:43.339958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:43.339958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:43.339958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:43.339958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:43.339958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:43.339958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:43.339958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:43.339958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:43.339958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:43.339958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:43.339958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:43.341849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:43.341849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:43.341849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:43.341849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:43.341849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:43.341849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:43.341849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:43.341849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:43.341849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:43.341849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:43.341849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:43.341849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:43.341849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:43.341849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:43.341849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:43.341849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:43.341849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:43.343612 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:43.343612 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:43.343612 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:43.350507 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 361/719] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 407 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:43.350507 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9603aa4-157d-4c2d-aa4f-08b8144db2a8 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:43.400092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:43.541949 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d603abd7-bacd-4ebd-8234-ca882b4ae7a3 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:43.628599 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9603aa4-157d-4c2d-aa4f-08b8144db2a8 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:43.632197 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03dbcf53-5b06-40d5-befe-489eb9d0244b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:43.633198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dbcf53-5b06-40d5-befe-489eb9d0244b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:43.633702 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dbcf53-5b06-40d5-befe-489eb9d0244b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:43.634175 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dbcf53-5b06-40d5-befe-489eb9d0244b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:43.647172 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03dbcf53-5b06-40d5-befe-489eb9d0244b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:43.649150 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03dbcf53-5b06-40d5-befe-489eb9d0244b None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:43.665163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-03dbcf53-5b06-40d5-befe-489eb9d0244b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:43.665163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:43.665163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:43.665163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:43.665163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:43.665163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:43.665163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:43.665163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:43.665163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:43.665163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:43.665163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:43.665163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:43.665163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:43.665163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:43.665163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:43.665163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:43.666139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:43.666139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:43.666139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:43.666139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:43.666139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:43.666139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:43.666139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:43.666139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:43.666139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:43.666139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:43.666139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:43.666139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:43.666139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:43.666139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:43.666139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:43.666139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:43.666139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:43.667608 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:43.667608 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:43.667608 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:43.669672 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 359/720] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 394 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:43:43.671333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387d85b2-5e1f-4adf-8be2-2b54cc0401b8 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:43.671612 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b9a00d-551d-4a50-8736-612cbe242382 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:43.671963 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387d85b2-5e1f-4adf-8be2-2b54cc0401b8 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:43.672374 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387d85b2-5e1f-4adf-8be2-2b54cc0401b8 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:43.672535 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b9a00d-551d-4a50-8736-612cbe242382 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:43.672922 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b9a00d-551d-4a50-8736-612cbe242382 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:43.673304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b9a00d-551d-4a50-8736-612cbe242382 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:43.674079 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b9a00d-551d-4a50-8736-612cbe242382 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:43.675889 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b9a00d-551d-4a50-8736-612cbe242382 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:43.679153 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b9a00d-551d-4a50-8736-612cbe242382 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:43.679449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 360/721] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:43 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:43.680385 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387d85b2-5e1f-4adf-8be2-2b54cc0401b8 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:43.693011 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 362/722] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 344 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:43.694796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e6f23d-033a-495e-8fcb-6bef06d6d445 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:43.695301 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e6f23d-033a-495e-8fcb-6bef06d6d445 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:43.695785 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e6f23d-033a-495e-8fcb-6bef06d6d445 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:43.719316 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-20e6f23d-033a-495e-8fcb-6bef06d6d445 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:43.721643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-20e6f23d-033a-495e-8fcb-6bef06d6d445 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:43.783455 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:43.807912 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387d85b2-5e1f-4adf-8be2-2b54cc0401b8 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:43.811958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:43.818810 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 363/723] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:43:43.820052 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e6f23d-033a-495e-8fcb-6bef06d6d445 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:43.870750 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82acaa2a-a1a0-4af3-9a54-52e087b60327 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:43.872594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82acaa2a-a1a0-4af3-9a54-52e087b60327 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:43.873733 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82acaa2a-a1a0-4af3-9a54-52e087b60327 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:43.874352 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82acaa2a-a1a0-4af3-9a54-52e087b60327 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:43.876273 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82acaa2a-a1a0-4af3-9a54-52e087b60327 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:43.876975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82acaa2a-a1a0-4af3-9a54-52e087b60327 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:43.879492 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82acaa2a-a1a0-4af3-9a54-52e087b60327 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:43.906313 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 361/724] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:43.913886 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82acaa2a-a1a0-4af3-9a54-52e087b60327 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:44.042756 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e6f23d-033a-495e-8fcb-6bef06d6d445 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:44.059707 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82acaa2a-a1a0-4af3-9a54-52e087b60327 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:44.096813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f11e58-436c-4a73-a594-a33e1856491e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:44.097557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f11e58-436c-4a73-a594-a33e1856491e None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:44.097909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f11e58-436c-4a73-a594-a33e1856491e None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:44.098252 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f11e58-436c-4a73-a594-a33e1856491e None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:44.106597 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f11e58-436c-4a73-a594-a33e1856491e None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:44.108066 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f11e58-436c-4a73-a594-a33e1856491e None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:44.121159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-19f11e58-436c-4a73-a594-a33e1856491e None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 18 00:43:44.121159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:44.121159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:44.121159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:44.121159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:44.121159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:44.121159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:44.121159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:44.121159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:44.121159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:44.121159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:44.121159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:44.121159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:43:44.121159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:43:44.121159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:43:44.121159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:43:44.122239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:44.122239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:44.122239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:43:44.122239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:44.122239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:43:44.122239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:44.122239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:44.122239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:44.122239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:44.122239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:44.122239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:44.122239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:44.122239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:44.122239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:44.122239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:44.122239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:44.122239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:44.122239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:44.123217 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:44.123217 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:44.123217 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:43:44.123217 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:43:44.123217 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:43:44.123217 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:43:44.123217 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:43:44.123217 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:43:44.123217 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 18 00:43:44.123217 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:44.123217 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 362/725] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Sep 18 00:43:43 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:43:44.128591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492b6cd7-0dfe-46c3-a4e5-c1d483bde036 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:44.129444 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492b6cd7-0dfe-46c3-a4e5-c1d483bde036 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:44.129720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492b6cd7-0dfe-46c3-a4e5-c1d483bde036 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:44.130871 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f11e58-436c-4a73-a594-a33e1856491e None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:44.133871 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492b6cd7-0dfe-46c3-a4e5-c1d483bde036 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:44.153031 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492b6cd7-0dfe-46c3-a4e5-c1d483bde036 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:44.155163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492b6cd7-0dfe-46c3-a4e5-c1d483bde036 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:44.166680 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:44.172548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-492b6cd7-0dfe-46c3-a4e5-c1d483bde036 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:44.172548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:44.172548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:44.172548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:44.172548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:44.172548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:44.172548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:44.172548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:44.172548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:44.172548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:44.172548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:44.172548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:44.172548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:43:44.172548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:43:44.172548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:43:44.173494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:43:44.173494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:44.173494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:44.173494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:44.173494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:44.173494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:44.173494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:44.173494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:44.173494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:44.173494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:44.173494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:44.173494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:44.173494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:44.173494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:44.173494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:44.173494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:44.173494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:44.174381 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:43:44.174381 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:43:44.174381 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:43:44.174381 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:43:44.174381 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:43:44.174381 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:43:44.174381 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:44.174381 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:44.178733 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 364/726] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 354 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:44.179670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492b6cd7-0dfe-46c3-a4e5-c1d483bde036 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:44.226469 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:44.276423 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f11e58-436c-4a73-a594-a33e1856491e None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:44.305382 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492b6cd7-0dfe-46c3-a4e5-c1d483bde036 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:44.315627 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8de663c-7d0e-47a7-8b51-f22ede732594 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:44.316538 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8de663c-7d0e-47a7-8b51-f22ede732594 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:44.316994 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8de663c-7d0e-47a7-8b51-f22ede732594 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:44.317461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8de663c-7d0e-47a7-8b51-f22ede732594 None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:44.318503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8de663c-7d0e-47a7-8b51-f22ede732594 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:44.318984 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8de663c-7d0e-47a7-8b51-f22ede732594 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:44.320752 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8de663c-7d0e-47a7-8b51-f22ede732594 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:44.342010 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 363/727] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Sep 18 00:43:44 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:44.343540 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8de663c-7d0e-47a7-8b51-f22ede732594 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:44.353073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1571aeba-14d3-46d9-95d4-598e4f36446e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:44.354039 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1571aeba-14d3-46d9-95d4-598e4f36446e None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:44.358837 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1571aeba-14d3-46d9-95d4-598e4f36446e None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:44.359330 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1571aeba-14d3-46d9-95d4-598e4f36446e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:44.372275 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1571aeba-14d3-46d9-95d4-598e4f36446e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:44.379179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-1571aeba-14d3-46d9-95d4-598e4f36446e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:43:44.379179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:44.379179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:44.379179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:44.379179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:44.379179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:44.379179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:44.379179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:44.379179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:44.379179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:44.379179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:44.379179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:44.379179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:44.379179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:44.379179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:44.379179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:44.380165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:44.380165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:44.380165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:44.380165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:44.380165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:44.380165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:44.380165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:44.380165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:44.380165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:44.380165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:44.380165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:44.380165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:44.380165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:44.380165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:44.380165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:44.380165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:44.380995 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:44.380995 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:44.380995 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:44.380995 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:44.380995 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:44.380995 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:44.380995 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:44.380995 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:44.380995 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:44.380995 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:44.380995 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:43:44.380995 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:44.383331 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 365/728] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:43:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:44.384525 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1571aeba-14d3-46d9-95d4-598e4f36446e None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:44.495649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:44.532236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1571aeba-14d3-46d9-95d4-598e4f36446e None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:44.534080 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8de663c-7d0e-47a7-8b51-f22ede732594 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:44.534577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:44.579459 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02167750-b50e-4dfe-80b7-9f6c0c3aadfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:44.580474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02167750-b50e-4dfe-80b7-9f6c0c3aadfe None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:44.581253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02167750-b50e-4dfe-80b7-9f6c0c3aadfe None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:44.581730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02167750-b50e-4dfe-80b7-9f6c0c3aadfe None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:44.602039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f99995-bd6e-4156-9122-d662a96830a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:44.603159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f99995-bd6e-4156-9122-d662a96830a9 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:44.603631 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f99995-bd6e-4156-9122-d662a96830a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:44.604122 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f99995-bd6e-4156-9122-d662a96830a9 None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:44.605519 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f99995-bd6e-4156-9122-d662a96830a9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:44.606142 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f99995-bd6e-4156-9122-d662a96830a9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:44.608970 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f99995-bd6e-4156-9122-d662a96830a9 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:44.637318 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 364/729] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:43:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:44.637318 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f99995-bd6e-4156-9122-d662a96830a9 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:44.681347 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02167750-b50e-4dfe-80b7-9f6c0c3aadfe None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:44.684306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02167750-b50e-4dfe-80b7-9f6c0c3aadfe None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:44.815740 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 366/730] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:43:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 18 00:43:44.817086 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02167750-b50e-4dfe-80b7-9f6c0c3aadfe None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:44.820789 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:44.832521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f99995-bd6e-4156-9122-d662a96830a9 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:44.886708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b34d8a9-0828-4872-bd7f-6b1b8fcc3de7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:44.887518 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b34d8a9-0828-4872-bd7f-6b1b8fcc3de7 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:44.887807 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b34d8a9-0828-4872-bd7f-6b1b8fcc3de7 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:44.888077 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b34d8a9-0828-4872-bd7f-6b1b8fcc3de7 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:44.889172 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b34d8a9-0828-4872-bd7f-6b1b8fcc3de7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:44.889692 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b34d8a9-0828-4872-bd7f-6b1b8fcc3de7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:44.891091 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b34d8a9-0828-4872-bd7f-6b1b8fcc3de7 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:44.902055 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 365/731] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:43:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:44.909187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb8aeaf-66b9-43c4-9343-f9925ef2a1bc None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:44.909638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb8aeaf-66b9-43c4-9343-f9925ef2a1bc None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:44.910081 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb8aeaf-66b9-43c4-9343-f9925ef2a1bc None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:44.913636 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 366/732] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:44 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:44.918788 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb8aeaf-66b9-43c4-9343-f9925ef2a1bc None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:44.920455 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:44.960346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02167750-b50e-4dfe-80b7-9f6c0c3aadfe None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:45.001673 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b864ed36-b93a-41bd-8fd4-1e4cfc4895bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:45.002734 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b864ed36-b93a-41bd-8fd4-1e4cfc4895bd None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:45.003054 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b864ed36-b93a-41bd-8fd4-1e4cfc4895bd None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:45.003323 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b864ed36-b93a-41bd-8fd4-1e4cfc4895bd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:45.004183 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b864ed36-b93a-41bd-8fd4-1e4cfc4895bd None admin] RBAC: Adding query filter params (role.id=1e04c9fea65244d28e1e59fc4c64f32f, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=a59d73fd45a54651930d52300704fdb0) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:45.004481 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b864ed36-b93a-41bd-8fd4-1e4cfc4895bd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:45.005920 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b864ed36-b93a-41bd-8fd4-1e4cfc4895bd None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:45.013678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 367/733] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:43:44 2022] GET /identity/v3/role_assignments?user.id=a59d73fd45a54651930d52300704fdb0&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=1e04c9fea65244d28e1e59fc4c64f32f => generated 259 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:45.021981 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b864ed36-b93a-41bd-8fd4-1e4cfc4895bd None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:45.062243 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb8aeaf-66b9-43c4-9343-f9925ef2a1bc None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:45.105416 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8114edf1-28be-476d-ad34-b7898eca7867 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:45.105988 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8114edf1-28be-476d-ad34-b7898eca7867 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:45.106276 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8114edf1-28be-476d-ad34-b7898eca7867 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:45.106574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8114edf1-28be-476d-ad34-b7898eca7867 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:45.120088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8114edf1-28be-476d-ad34-b7898eca7867 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:45.122171 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8114edf1-28be-476d-ad34-b7898eca7867 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:45.141734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-8114edf1-28be-476d-ad34-b7898eca7867 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:45.141734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:45.141734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:45.141734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:45.141734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:45.141734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:45.141734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:45.141734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:45.141734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:45.141734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:45.141734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:45.141734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:45.141734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:45.141734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:45.141734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:45.141734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:45.143594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:45.143594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:45.143594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:45.143594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:45.143594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:45.143594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:45.143594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:45.143594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:45.143594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:45.143594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:45.143594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:45.143594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:45.143594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:45.143594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:45.143594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:45.143594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:45.143594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:45.145222 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:45.145222 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:45.145222 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:45.145222 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 367/734] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:45.146557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98132520-a51d-431d-8b76-646ee6b96381 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:45.147061 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98132520-a51d-431d-8b76-646ee6b96381 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:45.147547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98132520-a51d-431d-8b76-646ee6b96381 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:45.176484 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-98132520-a51d-431d-8b76-646ee6b96381 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:45.177975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-98132520-a51d-431d-8b76-646ee6b96381 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:45.179148 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:45.237631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b864ed36-b93a-41bd-8fd4-1e4cfc4895bd None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:45.255419 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:45.292380 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 368/735] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:43:45.292380 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bef487-5496-4624-8f7a-8e7ff7f8e877 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:45.292380 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bef487-5496-4624-8f7a-8e7ff7f8e877 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:45.292380 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bef487-5496-4624-8f7a-8e7ff7f8e877 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:45.292380 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 369/736] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:45 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:45.292380 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bef487-5496-4624-8f7a-8e7ff7f8e877 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:45.296795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2072a15-950b-4b66-aaf8-63759e6971c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:45.297847 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2072a15-950b-4b66-aaf8-63759e6971c7 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:45.298319 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2072a15-950b-4b66-aaf8-63759e6971c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:45.298822 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2072a15-950b-4b66-aaf8-63759e6971c7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:45.319556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2072a15-950b-4b66-aaf8-63759e6971c7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:45.324803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2072a15-950b-4b66-aaf8-63759e6971c7 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:45.361037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-e2072a15-950b-4b66-aaf8-63759e6971c7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:45.361037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:45.361037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:45.361037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:45.361037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:45.361037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:45.361037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:45.361037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:45.361037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:45.361037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:45.361037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:45.361037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:45.361037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:45.361037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:45.361037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:45.361037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:45.362009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:45.362009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:45.362009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:45.362009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:45.362009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:45.362009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:45.362009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:45.362009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:45.362009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:45.362009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:45.362009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:45.362009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:45.362009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:45.362009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:45.362009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:45.362009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:45.362009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:45.362931 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:45.362931 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:45.362931 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:45.364516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 368/737] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 350 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:45.366939 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bf7625-b33a-418b-bfbe-8b3a3fc8cc6b None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:45.367409 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bf7625-b33a-418b-bfbe-8b3a3fc8cc6b None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:45.367864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bf7625-b33a-418b-bfbe-8b3a3fc8cc6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:45.407093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-b6bf7625-b33a-418b-bfbe-8b3a3fc8cc6b None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:45.408435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-b6bf7625-b33a-418b-bfbe-8b3a3fc8cc6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:45.448538 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bef487-5496-4624-8f7a-8e7ff7f8e877 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:45.500061 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec15391-30c2-46dc-acd1-c54c23ddfece None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:45.500699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec15391-30c2-46dc-acd1-c54c23ddfece None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:45.500984 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec15391-30c2-46dc-acd1-c54c23ddfece None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:45.501263 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec15391-30c2-46dc-acd1-c54c23ddfece None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:45.501972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec15391-30c2-46dc-acd1-c54c23ddfece None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:45.502291 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec15391-30c2-46dc-acd1-c54c23ddfece None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:45.503752 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec15391-30c2-46dc-acd1-c54c23ddfece None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:45.523987 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 370/738] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:45.525617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec15391-30c2-46dc-acd1-c54c23ddfece None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:45.547248 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 369/739] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 18 00:43:45.548734 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6bf7625-b33a-418b-bfbe-8b3a3fc8cc6b None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:45.614856 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:45.632838 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:45.681756 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec15391-30c2-46dc-acd1-c54c23ddfece None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:45.697536 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6bf7625-b33a-418b-bfbe-8b3a3fc8cc6b None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:45.728287 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af383919-085a-43a3-a711-f086824138b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:45.728920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af383919-085a-43a3-a711-f086824138b7 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:45.729194 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af383919-085a-43a3-a711-f086824138b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:45.729506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af383919-085a-43a3-a711-f086824138b7 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:45.730163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af383919-085a-43a3-a711-f086824138b7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:45.730480 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af383919-085a-43a3-a711-f086824138b7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:45.732194 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af383919-085a-43a3-a711-f086824138b7 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:45.744318 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730ea84a-d1fb-4c59-934e-311676d68052 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:45.745186 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730ea84a-d1fb-4c59-934e-311676d68052 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:45.745526 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730ea84a-d1fb-4c59-934e-311676d68052 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:45.745869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730ea84a-d1fb-4c59-934e-311676d68052 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:45.746585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 371/740] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:45.747437 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af383919-085a-43a3-a711-f086824138b7 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:45.754619 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730ea84a-d1fb-4c59-934e-311676d68052 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:45.756247 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730ea84a-d1fb-4c59-934e-311676d68052 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:45.770030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-730ea84a-d1fb-4c59-934e-311676d68052 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:45.770030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:45.770030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:45.770030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:45.770030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:45.770030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:45.770030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:45.770030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:45.770030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:45.770030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:45.770030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:45.770030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:45.770030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:43:45.770030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:43:45.770030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:43:45.771001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:43:45.771001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:45.771001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:45.771001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:45.771001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:45.771001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:45.771001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:45.771001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:45.771001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:45.771001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:45.771001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:45.771001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:45.771001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:45.771001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:45.771001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:45.771001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:45.771001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:45.771885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:43:45.771885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:43:45.771885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:43:45.771885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:43:45.771885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:43:45.771885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:43:45.771885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:45.771885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:45.777053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 370/741] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:45.778204 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730ea84a-d1fb-4c59-934e-311676d68052 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:45.869347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af383919-085a-43a3-a711-f086824138b7 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:45.882311 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:45.900276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730ea84a-d1fb-4c59-934e-311676d68052 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:45.913532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:45.934330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfe608e-3127-4e35-bbaa-b999785b9863 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:45.935750 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfe608e-3127-4e35-bbaa-b999785b9863 None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:45.936442 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfe608e-3127-4e35-bbaa-b999785b9863 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:45.937121 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfe608e-3127-4e35-bbaa-b999785b9863 None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:45.938592 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfe608e-3127-4e35-bbaa-b999785b9863 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:45.940934 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d3a2be-387d-4009-8222-f234e752fdc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:45.941576 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d3a2be-387d-4009-8222-f234e752fdc1 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:45.941852 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d3a2be-387d-4009-8222-f234e752fdc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:45.942125 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d3a2be-387d-4009-8222-f234e752fdc1 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:45.942405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfe608e-3127-4e35-bbaa-b999785b9863 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:45.951219 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d3a2be-387d-4009-8222-f234e752fdc1 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:45.952745 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d3a2be-387d-4009-8222-f234e752fdc1 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:45.965977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-43d3a2be-387d-4009-8222-f234e752fdc1 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 18 00:43:45.965977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:45.965977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:45.965977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:45.965977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:45.965977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:45.965977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:45.965977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:45.965977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:45.965977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:45.965977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:45.965977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:45.965977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:43:45.965977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:43:45.965977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:43:45.965977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:43:45.967912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:45.967912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:45.967912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:43:45.967912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:45.967912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:43:45.967912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:45.967912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:45.967912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:45.967912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:45.967912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:45.967912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:45.967912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:45.967912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:45.967912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:45.967912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:45.967912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:45.967912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:45.967912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:45.969703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:45.969703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:45.969703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:43:45.969703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:43:45.969703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:43:45.969703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:43:45.969703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:43:45.969703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:43:45.969703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 18 00:43:45.969703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:45.969703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 371/742] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:43:45 2022] GET /identity/v3/users/swift => generated 83 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:45.969703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d3a2be-387d-4009-8222-f234e752fdc1 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:45.978503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 372/743] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 18 00:43:45 2022] POST /identity/v3/services => generated 251 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:43:45.978503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfe608e-3127-4e35-bbaa-b999785b9863 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:46.131248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfe608e-3127-4e35-bbaa-b999785b9863 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:46.137886 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d3a2be-387d-4009-8222-f234e752fdc1 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:46.179319 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4770bee-2b36-4b46-b951-8a80ae697d32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:46.180087 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4770bee-2b36-4b46-b951-8a80ae697d32 None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:46.180402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4770bee-2b36-4b46-b951-8a80ae697d32 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:46.180748 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4770bee-2b36-4b46-b951-8a80ae697d32 None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:46.181488 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4770bee-2b36-4b46-b951-8a80ae697d32 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:46.183575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4770bee-2b36-4b46-b951-8a80ae697d32 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:46.184839 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63e1590-9afc-4a18-a8a4-571d9de56b50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:46.186098 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63e1590-9afc-4a18-a8a4-571d9de56b50 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:46.186594 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63e1590-9afc-4a18-a8a4-571d9de56b50 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:46.187086 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63e1590-9afc-4a18-a8a4-571d9de56b50 None admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:46.188438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63e1590-9afc-4a18-a8a4-571d9de56b50 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:46.189598 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63e1590-9afc-4a18-a8a4-571d9de56b50 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:46.191599 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63e1590-9afc-4a18-a8a4-571d9de56b50 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:46.203837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 373/744] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 18 00:43:45 2022] POST /identity/v3/services => generated 240 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:43:46.205304 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4770bee-2b36-4b46-b951-8a80ae697d32 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:46.223772 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 372/745] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:43:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:46.225985 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63e1590-9afc-4a18-a8a4-571d9de56b50 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:46.272538 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:46.294326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:46.380858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4770bee-2b36-4b46-b951-8a80ae697d32 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:46.425074 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622a3b4a-77d5-43c0-bed5-a122a12e353f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:46.425890 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622a3b4a-77d5-43c0-bed5-a122a12e353f None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:46.426172 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622a3b4a-77d5-43c0-bed5-a122a12e353f None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:46.426442 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622a3b4a-77d5-43c0-bed5-a122a12e353f None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:46.427234 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622a3b4a-77d5-43c0-bed5-a122a12e353f None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:46.427543 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622a3b4a-77d5-43c0-bed5-a122a12e353f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:46.429037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622a3b4a-77d5-43c0-bed5-a122a12e353f None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:46.441070 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63e1590-9afc-4a18-a8a4-571d9de56b50 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:46.455420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 374/746] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Sep 18 00:43:46 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:46.463902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622a3b4a-77d5-43c0-bed5-a122a12e353f None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:46.513041 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c461e0e-4bed-4f6b-af0a-1b18fd9f9b5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:46.514106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c461e0e-4bed-4f6b-af0a-1b18fd9f9b5a None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:46.514627 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c461e0e-4bed-4f6b-af0a-1b18fd9f9b5a None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:46.515115 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c461e0e-4bed-4f6b-af0a-1b18fd9f9b5a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:46.516569 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c461e0e-4bed-4f6b-af0a-1b18fd9f9b5a None admin] RBAC: Adding query filter params (role.id=81523e544c7e4993af4d1e06ce3754bc, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=3e31dce69c344c9aa4a55f31b9c735df) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:46.517112 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c461e0e-4bed-4f6b-af0a-1b18fd9f9b5a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:46.521709 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c461e0e-4bed-4f6b-af0a-1b18fd9f9b5a None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:46.533259 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 373/747] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:43:46 2022] GET /identity/v3/role_assignments?user.id=3e31dce69c344c9aa4a55f31b9c735df&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=81523e544c7e4993af4d1e06ce3754bc => generated 259 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:46.612186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:46.666563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622a3b4a-77d5-43c0-bed5-a122a12e353f None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:46.743900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d97e33c-f264-42f2-8cb5-bc17e9708009 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:46.744864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d97e33c-f264-42f2-8cb5-bc17e9708009 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:46.745561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d97e33c-f264-42f2-8cb5-bc17e9708009 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:46.746046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d97e33c-f264-42f2-8cb5-bc17e9708009 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:46.888767 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d97e33c-f264-42f2-8cb5-bc17e9708009 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:46.891409 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d97e33c-f264-42f2-8cb5-bc17e9708009 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:47.043164 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:47.090840 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 375/748] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:43:46 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 627 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 18 00:43:47.097133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c461e0e-4bed-4f6b-af0a-1b18fd9f9b5a None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:47.250739 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:47.262186 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c461e0e-4bed-4f6b-af0a-1b18fd9f9b5a None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:47.310162 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85bdd078-b23c-4c1e-8036-b5225a92976e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:47.310846 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bdd078-b23c-4c1e-8036-b5225a92976e None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:47.311124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bdd078-b23c-4c1e-8036-b5225a92976e None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:47.311398 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bdd078-b23c-4c1e-8036-b5225a92976e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:47.323933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85bdd078-b23c-4c1e-8036-b5225a92976e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:47.325408 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85bdd078-b23c-4c1e-8036-b5225a92976e None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:47.345801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-85bdd078-b23c-4c1e-8036-b5225a92976e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:47.345801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:47.345801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:47.345801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:47.345801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:47.345801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:47.345801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:47.345801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:47.345801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:47.345801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:47.345801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:47.345801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:47.345801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:47.345801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:47.345801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:47.345801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:47.347195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:47.347195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:47.347195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:47.347195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:47.347195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:47.347195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:47.347195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:47.347195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:47.347195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:47.347195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:47.347195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:47.347195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:47.347195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:47.347195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:47.347195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:47.347195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:47.347195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:47.348261 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:47.348261 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:47.348261 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:47.351204 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 374/749] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:47.357399 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d97e33c-f264-42f2-8cb5-bc17e9708009 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:47.515636 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d97e33c-f264-42f2-8cb5-bc17e9708009 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:47.571056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d01ddf-e74b-4f07-8487-b71e28f549f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:47.571610 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d01ddf-e74b-4f07-8487-b71e28f549f2 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:47.571893 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d01ddf-e74b-4f07-8487-b71e28f549f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:47.572167 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d01ddf-e74b-4f07-8487-b71e28f549f2 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:47.572848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d01ddf-e74b-4f07-8487-b71e28f549f2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:47.573669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d01ddf-e74b-4f07-8487-b71e28f549f2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:47.575148 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d01ddf-e74b-4f07-8487-b71e28f549f2 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:47.588355 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 376/750] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:47.594088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85bdd078-b23c-4c1e-8036-b5225a92976e None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:47.790353 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85bdd078-b23c-4c1e-8036-b5225a92976e None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:47.864564 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ca3ba2-9060-4b73-97e2-7df6e7d9ecd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:47.865441 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ca3ba2-9060-4b73-97e2-7df6e7d9ecd1 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:47.865864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ca3ba2-9060-4b73-97e2-7df6e7d9ecd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:47.866270 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ca3ba2-9060-4b73-97e2-7df6e7d9ecd1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:47.878789 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ca3ba2-9060-4b73-97e2-7df6e7d9ecd1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:47.881789 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ca3ba2-9060-4b73-97e2-7df6e7d9ecd1 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:47.911199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-27ca3ba2-9060-4b73-97e2-7df6e7d9ecd1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:47.911199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:47.911199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:47.911199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:47.911199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:47.911199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:47.911199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:47.911199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:47.911199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:47.911199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:47.911199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:47.911199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:47.911199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:43:47.911199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:43:47.911199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:43:47.912137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:43:47.912137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:47.912137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:47.912137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:47.912137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:47.912137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:47.912137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:47.912137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:47.912137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:47.912137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:47.912137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:47.912137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:47.912137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:47.912137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:47.912137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:47.912137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:47.912137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:47.913024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:43:47.913024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:43:47.913024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:43:47.913024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:43:47.913024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:43:47.913024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:43:47.913024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:47.913024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:47.913915 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 375/751] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 320 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:47.923902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d01ddf-e74b-4f07-8487-b71e28f549f2 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:48.013099 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:48.080862 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e07ad8-d36f-44a0-b770-96c23d926d83 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:48.081321 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e07ad8-d36f-44a0-b770-96c23d926d83 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:48.081787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e07ad8-d36f-44a0-b770-96c23d926d83 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:48.084260 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 376/752] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:48 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:48.105715 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8ecb02-4a8d-4414-8a52-f714aa7a1284 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:48.106181 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8ecb02-4a8d-4414-8a52-f714aa7a1284 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:48.106699 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8ecb02-4a8d-4414-8a52-f714aa7a1284 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:48.126575 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:48.151762 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-1c8ecb02-4a8d-4414-8a52-f714aa7a1284 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:48.157016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-1c8ecb02-4a8d-4414-8a52-f714aa7a1284 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:48.162510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d01ddf-e74b-4f07-8487-b71e28f549f2 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:48.247200 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 377/753] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:48 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:43:48.265804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7897d1bc-cef8-4650-801e-2dd6c1f85116 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:48.266918 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7897d1bc-cef8-4650-801e-2dd6c1f85116 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:48.267258 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7897d1bc-cef8-4650-801e-2dd6c1f85116 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:48.267659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7897d1bc-cef8-4650-801e-2dd6c1f85116 None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:48.268977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7897d1bc-cef8-4650-801e-2dd6c1f85116 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:48.269503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7897d1bc-cef8-4650-801e-2dd6c1f85116 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:48.280490 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7897d1bc-cef8-4650-801e-2dd6c1f85116 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:48.320739 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 377/754] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:43:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 398 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:48.330540 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8ecb02-4a8d-4414-8a52-f714aa7a1284 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:48.472004 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8ecb02-4a8d-4414-8a52-f714aa7a1284 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:48.517605 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccd6628-9cd0-4247-8746-cb8e6725ce01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:48.518253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccd6628-9cd0-4247-8746-cb8e6725ce01 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:48.519037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccd6628-9cd0-4247-8746-cb8e6725ce01 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:48.519322 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccd6628-9cd0-4247-8746-cb8e6725ce01 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:48.520092 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccd6628-9cd0-4247-8746-cb8e6725ce01 None admin] RBAC: Adding query filter params (role.id=1e04c9fea65244d28e1e59fc4c64f32f, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=6230ed7ce0eb43e9b2b8530a28eb6299) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:48.520401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccd6628-9cd0-4247-8746-cb8e6725ce01 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:48.521907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccd6628-9cd0-4247-8746-cb8e6725ce01 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:48.527201 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:48.533393 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 378/755] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:43:48 2022] GET /identity/v3/role_assignments?user.id=6230ed7ce0eb43e9b2b8530a28eb6299&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=1e04c9fea65244d28e1e59fc4c64f32f => generated 259 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:43:48.602945 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7897d1bc-cef8-4650-801e-2dd6c1f85116 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:48.806042 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:48.816798 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7897d1bc-cef8-4650-801e-2dd6c1f85116 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:48.869418 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb21f1ad-27c0-41d0-b808-092c725f8c43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:48.870117 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb21f1ad-27c0-41d0-b808-092c725f8c43 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:48.870430 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb21f1ad-27c0-41d0-b808-092c725f8c43 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:48.870729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb21f1ad-27c0-41d0-b808-092c725f8c43 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:48.882182 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb21f1ad-27c0-41d0-b808-092c725f8c43 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:48.887849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-fb21f1ad-27c0-41d0-b808-092c725f8c43 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:43:48.887849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:48.887849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:48.887849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:48.887849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:48.887849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:48.887849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:48.887849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:48.887849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:48.887849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:48.887849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:48.887849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:48.887849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:48.887849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:48.887849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:48.887849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:48.888797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:48.888797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:48.888797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:48.888797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:48.888797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:48.888797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:48.888797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:48.888797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:48.888797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:48.888797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:48.888797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:48.888797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:48.888797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:48.888797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:48.888797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:48.888797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:48.889612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:48.889612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:48.889612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:48.889612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:48.889612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:48.889612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:48.889612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:48.889612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:48.889612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:48.889612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:48.889612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:43:48.889612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:48.889612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 378/756] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:43:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 287 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:43:48.897448 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccd6628-9cd0-4247-8746-cb8e6725ce01 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:49.161294 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccd6628-9cd0-4247-8746-cb8e6725ce01 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:49.230092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8307f2-8164-48e5-8a59-9f5b2fe90fea None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:49.230939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8307f2-8164-48e5-8a59-9f5b2fe90fea None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:49.230939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8307f2-8164-48e5-8a59-9f5b2fe90fea None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:49.233043 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 379/757] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:49.238076 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073ac6d5-0507-4e89-81a2-13fc13fbc215 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:49.239576 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073ac6d5-0507-4e89-81a2-13fc13fbc215 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:49.240124 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfd50a8-e372-4805-a724-61f1bc7accfc None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:49.240415 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfd50a8-e372-4805-a724-61f1bc7accfc None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:49.240686 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfd50a8-e372-4805-a724-61f1bc7accfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:49.241091 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073ac6d5-0507-4e89-81a2-13fc13fbc215 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:49.241805 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073ac6d5-0507-4e89-81a2-13fc13fbc215 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:49.243399 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073ac6d5-0507-4e89-81a2-13fc13fbc215 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:49.244273 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073ac6d5-0507-4e89-81a2-13fc13fbc215 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:49.247967 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073ac6d5-0507-4e89-81a2-13fc13fbc215 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:49.276903 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 379/758] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:43:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 380 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:43:49.286251 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073ac6d5-0507-4e89-81a2-13fc13fbc215 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:49.304467 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-8cfd50a8-e372-4805-a724-61f1bc7accfc None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:49.306996 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-8cfd50a8-e372-4805-a724-61f1bc7accfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:49.412736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:49.477273 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 380/759] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:49 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 18 00:43:49.480523 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bb2786-6305-46e9-824b-68b59d0cf142 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:49.481383 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bb2786-6305-46e9-824b-68b59d0cf142 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:49.482136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bb2786-6305-46e9-824b-68b59d0cf142 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:49.484908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 381/760] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:49 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:49.494490 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a7b9af-5eed-4039-96a1-52b50f456100 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:49.494490 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a7b9af-5eed-4039-96a1-52b50f456100 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:49.494490 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a7b9af-5eed-4039-96a1-52b50f456100 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:49.501278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073ac6d5-0507-4e89-81a2-13fc13fbc215 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:49.539767 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-33a7b9af-5eed-4039-96a1-52b50f456100 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:49.540868 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-33a7b9af-5eed-4039-96a1-52b50f456100 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:49.582258 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515a3581-39c2-414a-9ce9-087e02b056fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:49.583405 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515a3581-39c2-414a-9ce9-087e02b056fc None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:49.583878 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515a3581-39c2-414a-9ce9-087e02b056fc None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:49.584334 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515a3581-39c2-414a-9ce9-087e02b056fc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:49.629701 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515a3581-39c2-414a-9ce9-087e02b056fc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:49.632538 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515a3581-39c2-414a-9ce9-087e02b056fc None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:49.701738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-515a3581-39c2-414a-9ce9-087e02b056fc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:49.701738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:49.701738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:49.701738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:49.701738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:49.701738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:49.701738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:49.701738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:49.701738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:49.701738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:49.701738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:49.701738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:49.701738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:49.701738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:49.701738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:49.701738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:49.703559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:49.703559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:49.703559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:49.703559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:49.703559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:49.703559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:49.703559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:49.703559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:49.703559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:49.703559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:49.703559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:49.703559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:49.703559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:49.703559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:49.703559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:49.703559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:49.703559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:49.705350 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:49.705350 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:49.705350 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:49.709255 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 380/761] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 424 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:43:49.719920 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515a3581-39c2-414a-9ce9-087e02b056fc None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:49.771258 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 382/762] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:49 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 18 00:43:49.840412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a7b9af-5eed-4039-96a1-52b50f456100 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:49.900581 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:49.945469 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515a3581-39c2-414a-9ce9-087e02b056fc None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:49.973704 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:49.992433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a7b9af-5eed-4039-96a1-52b50f456100 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:50.020483 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0c1e79-38a5-428a-ae68-83e2f2cbbd75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:50.022157 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0c1e79-38a5-428a-ae68-83e2f2cbbd75 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:50.022988 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0c1e79-38a5-428a-ae68-83e2f2cbbd75 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:50.023805 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0c1e79-38a5-428a-ae68-83e2f2cbbd75 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:50.025474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0c1e79-38a5-428a-ae68-83e2f2cbbd75 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:50.026358 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0c1e79-38a5-428a-ae68-83e2f2cbbd75 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:50.029525 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0c1e79-38a5-428a-ae68-83e2f2cbbd75 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:50.033577 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db4f597-9729-410e-b9ad-75a7d44b2b9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:50.035447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db4f597-9729-410e-b9ad-75a7d44b2b9d None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:50.035447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db4f597-9729-410e-b9ad-75a7d44b2b9d None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:50.035746 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db4f597-9729-410e-b9ad-75a7d44b2b9d None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:50.036817 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db4f597-9729-410e-b9ad-75a7d44b2b9d None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:50.039510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db4f597-9729-410e-b9ad-75a7d44b2b9d None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:50.050005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-1db4f597-9729-410e-b9ad-75a7d44b2b9d None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 18 00:43:50.050005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:50.050005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:43:50.050005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:43:50.050005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:43:50.050005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:43:50.050005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:43:50.050005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:43:50.050005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 18 00:43:50.050005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:50.050005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:43:50.050005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:50.050005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:50.050005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:50.050005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:50.050005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:50.050005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:50.052046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:50.052046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:50.052046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:50.052046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:50.052046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:50.052046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:50.052046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:43:50.052046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:43:50.052046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:43:50.052046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:43:50.052046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:50.052046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:50.052046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:50.052046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:50.052046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:50.052046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:50.053535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:50.053535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:50.053535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:50.053535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:50.053535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:50.053535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:50.053535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:50.053535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:50.053535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:50.053535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:50.053535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:43:50.053535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:43:50.053535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 18 00:43:50.053535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:50.053535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 383/763] 10.222.0.42 () {56 vars in 1129 bytes} [Sun Sep 18 00:43:49 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:50.056191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00307997-6c58-401d-b89c-16ebc148f15f None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:50.056191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00307997-6c58-401d-b89c-16ebc148f15f None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:50.056191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00307997-6c58-401d-b89c-16ebc148f15f None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:50.060096 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 384/764] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:50.061113 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00307997-6c58-401d-b89c-16ebc148f15f None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:50.066303 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 381/765] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 347 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:50.069177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7bd7d8-5626-4281-bdcd-1cc57f3719c8 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:50.070040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7bd7d8-5626-4281-bdcd-1cc57f3719c8 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:50.070810 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7bd7d8-5626-4281-bdcd-1cc57f3719c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:50.106333 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-4d7bd7d8-5626-4281-bdcd-1cc57f3719c8 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:50.109304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-4d7bd7d8-5626-4281-bdcd-1cc57f3719c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:50.160028 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:50.184097 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00307997-6c58-401d-b89c-16ebc148f15f None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:50.221254 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7546e07-0d49-4af4-8e47-c48d0989b304 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:50.221871 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7546e07-0d49-4af4-8e47-c48d0989b304 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:50.222155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7546e07-0d49-4af4-8e47-c48d0989b304 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:50.222437 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7546e07-0d49-4af4-8e47-c48d0989b304 None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:50.223091 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7546e07-0d49-4af4-8e47-c48d0989b304 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:50.223493 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7546e07-0d49-4af4-8e47-c48d0989b304 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:50.224964 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7546e07-0d49-4af4-8e47-c48d0989b304 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:50.230949 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 385/766] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Sep 18 00:43:50 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:50.231895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7546e07-0d49-4af4-8e47-c48d0989b304 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:50.261156 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 382/767] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:50 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:43:50.263179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7bd7d8-5626-4281-bdcd-1cc57f3719c8 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:50.358839 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7546e07-0d49-4af4-8e47-c48d0989b304 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:50.396037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c5a3ec-b53e-4756-bb0e-4a4c4fa9ac57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:50.396689 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c5a3ec-b53e-4756-bb0e-4a4c4fa9ac57 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:50.396969 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c5a3ec-b53e-4756-bb0e-4a4c4fa9ac57 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:50.397244 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c5a3ec-b53e-4756-bb0e-4a4c4fa9ac57 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:50.405722 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c5a3ec-b53e-4756-bb0e-4a4c4fa9ac57 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:50.407217 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c5a3ec-b53e-4756-bb0e-4a4c4fa9ac57 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:50.419961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-b6c5a3ec-b53e-4756-bb0e-4a4c4fa9ac57 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 18 00:43:50.419961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:50.419961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:50.419961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:50.419961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:50.419961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:50.419961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:50.419961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:50.419961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:50.419961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:50.419961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:50.419961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:50.419961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:43:50.419961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:43:50.419961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:43:50.419961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:43:50.421831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:50.421831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:50.421831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:43:50.421831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:50.421831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:43:50.421831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:50.421831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:50.421831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:50.421831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:50.421831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:50.421831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:50.421831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:50.421831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:50.421831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:50.421831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:50.421831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:50.421831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:50.421831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:50.424574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:50.424574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:50.424574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:43:50.424574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:43:50.424574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:43:50.424574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:43:50.424574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:43:50.424574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:43:50.424574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 18 00:43:50.424574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:50.424574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 386/768] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Sep 18 00:43:50 2022] GET /identity/v3/users/nova => generated 82 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:50.424574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c5a3ec-b53e-4756-bb0e-4a4c4fa9ac57 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:50.427944 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:50.487500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7bd7d8-5626-4281-bdcd-1cc57f3719c8 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:50.491881 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:50.546410 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c5a3ec-b53e-4756-bb0e-4a4c4fa9ac57 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:50.568047 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0b535b-d173-458c-871c-cab95f16a986 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:50.569747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0b535b-d173-458c-871c-cab95f16a986 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:50.570750 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0b535b-d173-458c-871c-cab95f16a986 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:50.571534 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0b535b-d173-458c-871c-cab95f16a986 None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:50.573367 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0b535b-d173-458c-871c-cab95f16a986 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:50.574216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0b535b-d173-458c-871c-cab95f16a986 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:50.577577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0b535b-d173-458c-871c-cab95f16a986 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:50.584143 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdef0a8-9810-453b-878d-1a55b2dbc8a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:50.584819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdef0a8-9810-453b-878d-1a55b2dbc8a2 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:50.585081 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdef0a8-9810-453b-878d-1a55b2dbc8a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:50.585332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdef0a8-9810-453b-878d-1a55b2dbc8a2 None admin] PATH_INFO: `/v3/services/network` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:50.586018 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdef0a8-9810-453b-878d-1a55b2dbc8a2 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:50.587328 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdef0a8-9810-453b-878d-1a55b2dbc8a2 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:50.591631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 383/769] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Sep 18 00:43:50 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 329 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:50.593686 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0b535b-d173-458c-871c-cab95f16a986 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:50.597554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-5fdef0a8-9810-453b-878d-1a55b2dbc8a2 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 18 00:43:50.597554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:50.597554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:43:50.597554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:43:50.597554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:43:50.597554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:43:50.597554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:43:50.597554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:43:50.597554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 18 00:43:50.597554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:50.597554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:43:50.597554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:50.597554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:50.597554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:50.597554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:50.597554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:50.597554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:50.599479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:50.599479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:50.599479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:50.599479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:50.599479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:50.599479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:50.599479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:43:50.599479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:43:50.599479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:43:50.599479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:43:50.599479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:50.599479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:50.599479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:50.599479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:50.599479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:50.599479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:50.601156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:50.601156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:50.601156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:50.601156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:50.601156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:50.601156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:50.601156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:50.601156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:50.601156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:50.601156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:50.601156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:43:50.601156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:43:50.601156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 18 00:43:50.601156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:50.601156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 387/770] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:50 2022] GET /identity/v3/services/network => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:50.612273 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdef0a8-9810-453b-878d-1a55b2dbc8a2 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:50.688689 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:50.727234 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdef0a8-9810-453b-878d-1a55b2dbc8a2 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:50.765407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654bd799-d7da-41af-b50c-9b8b6214ed6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:50.765924 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654bd799-d7da-41af-b50c-9b8b6214ed6a None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:50.766206 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654bd799-d7da-41af-b50c-9b8b6214ed6a None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:50.766560 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654bd799-d7da-41af-b50c-9b8b6214ed6a None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:50.767202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654bd799-d7da-41af-b50c-9b8b6214ed6a None admin] RBAC: Adding query filter params (name=network) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:50.767537 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654bd799-d7da-41af-b50c-9b8b6214ed6a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:50.769271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654bd799-d7da-41af-b50c-9b8b6214ed6a None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:50.774343 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 388/771] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 18 00:43:50 2022] GET /identity/v3/services?name=network => generated 124 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:50.775230 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654bd799-d7da-41af-b50c-9b8b6214ed6a None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:50.814826 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0b535b-d173-458c-871c-cab95f16a986 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:50.883918 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d21666-94b2-4c14-bbdc-fdd490d9fbea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:50.885004 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d21666-94b2-4c14-bbdc-fdd490d9fbea None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:50.885488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d21666-94b2-4c14-bbdc-fdd490d9fbea None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:50.886039 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d21666-94b2-4c14-bbdc-fdd490d9fbea None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:50.887424 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d21666-94b2-4c14-bbdc-fdd490d9fbea None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:50.887966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d21666-94b2-4c14-bbdc-fdd490d9fbea None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:50.890782 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d21666-94b2-4c14-bbdc-fdd490d9fbea None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:50.923866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654bd799-d7da-41af-b50c-9b8b6214ed6a None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:50.930332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 384/772] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Sep 18 00:43:50 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 333 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:50.930332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d21666-94b2-4c14-bbdc-fdd490d9fbea None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:50.974848 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:50.981931 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:50.991974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba66587e-00bf-42e5-b0b9-6c1b30a85417 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:50.993120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba66587e-00bf-42e5-b0b9-6c1b30a85417 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:50.993608 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba66587e-00bf-42e5-b0b9-6c1b30a85417 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:50.994080 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba66587e-00bf-42e5-b0b9-6c1b30a85417 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:50.995489 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba66587e-00bf-42e5-b0b9-6c1b30a85417 None admin] RBAC: Adding query filter params (interface=public, service_id=f16bbc070e0745b2bcf04877f637f5f7, region_id=RegionOne) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:50.996038 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba66587e-00bf-42e5-b0b9-6c1b30a85417 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:50.998645 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba66587e-00bf-42e5-b0b9-6c1b30a85417 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:51.013072 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 389/773] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Sep 18 00:43:50 2022] GET /identity/v3/endpoints?service_id=f16bbc070e0745b2bcf04877f637f5f7&interface=public®ion_id=RegionOne => generated 194 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:51.014570 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba66587e-00bf-42e5-b0b9-6c1b30a85417 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:51.132812 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d21666-94b2-4c14-bbdc-fdd490d9fbea None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:51.173129 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba66587e-00bf-42e5-b0b9-6c1b30a85417 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:51.191281 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1226541-a196-4f3b-9534-468a6caec4df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:51.192252 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1226541-a196-4f3b-9534-468a6caec4df None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:51.192534 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1226541-a196-4f3b-9534-468a6caec4df None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:51.192857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1226541-a196-4f3b-9534-468a6caec4df None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:51.193635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1226541-a196-4f3b-9534-468a6caec4df None admin] RBAC: Adding query filter params (type=network) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:51.193939 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1226541-a196-4f3b-9534-468a6caec4df None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:51.195366 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1226541-a196-4f3b-9534-468a6caec4df None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:51.209333 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 385/774] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 18 00:43:50 2022] GET /identity/v3/services?type=network => generated 350 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:51.209333 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1226541-a196-4f3b-9534-468a6caec4df None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:51.240482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7046b24f-c8b0-4f5d-afc1-ce0734343cd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:51.241656 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7046b24f-c8b0-4f5d-afc1-ce0734343cd6 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:51.241656 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7046b24f-c8b0-4f5d-afc1-ce0734343cd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:51.242006 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7046b24f-c8b0-4f5d-afc1-ce0734343cd6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:51.259923 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7046b24f-c8b0-4f5d-afc1-ce0734343cd6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:51.270180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-7046b24f-c8b0-4f5d-afc1-ce0734343cd6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:51.270180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:51.270180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:51.270180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:51.270180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:51.270180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:51.270180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:51.270180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:51.270180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:51.270180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:51.270180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:51.270180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:51.270180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:51.270180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:51.270180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:51.270180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:51.275329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:51.275329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:51.275329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:51.275329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:51.275329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:51.275329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:51.275329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:51.275329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:51.275329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:51.275329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:51.275329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:51.275329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:51.275329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:51.275329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:51.275329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:51.275329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:51.276356 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:51.276356 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:51.276356 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:51.276356 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:51.276356 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:51.276356 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:51.276356 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:51.276356 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:51.276356 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:51.276356 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:51.276356 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:51.276356 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:51.276356 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 390/775] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:43:51 2022] GET /identity/v3/roles/service => generated 85 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:51.276356 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7046b24f-c8b0-4f5d-afc1-ce0734343cd6 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:51.371525 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:51.372340 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:51.397299 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1226541-a196-4f3b-9534-468a6caec4df None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:51.445951 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7046b24f-c8b0-4f5d-afc1-ce0734343cd6 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:51.468838 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06068c74-7ebb-4f3a-8cc9-96b3bdc1c878 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:51.469504 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06068c74-7ebb-4f3a-8cc9-96b3bdc1c878 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:51.469785 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06068c74-7ebb-4f3a-8cc9-96b3bdc1c878 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:51.470055 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06068c74-7ebb-4f3a-8cc9-96b3bdc1c878 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:51.482065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06068c74-7ebb-4f3a-8cc9-96b3bdc1c878 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:51.483610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06068c74-7ebb-4f3a-8cc9-96b3bdc1c878 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:51.501426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-06068c74-7ebb-4f3a-8cc9-96b3bdc1c878 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:51.501426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:51.501426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:51.501426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:51.501426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:51.501426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:51.501426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:51.501426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:51.501426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:51.501426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:51.501426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:51.501426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:51.501426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:51.501426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:51.501426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:51.501426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:51.503865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:51.503865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:51.503865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:51.503865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:51.503865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:51.503865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:51.503865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:51.503865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:51.503865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:51.503865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:51.503865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:51.503865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:51.503865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:51.503865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:51.503865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:51.503865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:51.503865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:51.505591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:51.505591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:51.505591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:51.505591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 386/776] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:51.505591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06068c74-7ebb-4f3a-8cc9-96b3bdc1c878 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:51.529287 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae07cb65-a3cf-435d-bd02-0c18be089906 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:51.529954 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae07cb65-a3cf-435d-bd02-0c18be089906 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:51.530227 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae07cb65-a3cf-435d-bd02-0c18be089906 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:51.530541 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae07cb65-a3cf-435d-bd02-0c18be089906 None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:51.531201 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae07cb65-a3cf-435d-bd02-0c18be089906 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:51.532653 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae07cb65-a3cf-435d-bd02-0c18be089906 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:51.538405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 391/777] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Sep 18 00:43:51 2022] GET /identity/v3/services => generated 772 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:51.539431 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae07cb65-a3cf-435d-bd02-0c18be089906 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:51.641693 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06068c74-7ebb-4f3a-8cc9-96b3bdc1c878 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:51.682161 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3729786f-9b12-4ab1-9067-81c43429b0f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:51.683287 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3729786f-9b12-4ab1-9067-81c43429b0f9 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:51.683564 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3729786f-9b12-4ab1-9067-81c43429b0f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:51.683833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3729786f-9b12-4ab1-9067-81c43429b0f9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:51.684553 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3729786f-9b12-4ab1-9067-81c43429b0f9 None admin] RBAC: Adding query filter params (interface=public, service_id=747605e706ef480ba6c8345809edebb6, region_id=RegionOne) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:51.684859 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3729786f-9b12-4ab1-9067-81c43429b0f9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:51.686253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3729786f-9b12-4ab1-9067-81c43429b0f9 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:51.696829 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 387/778] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Sep 18 00:43:51 2022] GET /identity/v3/endpoints?service_id=747605e706ef480ba6c8345809edebb6&interface=public®ion_id=RegionOne => generated 194 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:51.697838 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6666416a-d314-47bd-aba5-0948557a7ad7 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:51.698155 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6666416a-d314-47bd-aba5-0948557a7ad7 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:51.698545 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6666416a-d314-47bd-aba5-0948557a7ad7 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:51.702784 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 388/779] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:51.702784 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6666416a-d314-47bd-aba5-0948557a7ad7 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:51.707086 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:51.723660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae07cb65-a3cf-435d-bd02-0c18be089906 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:51.754832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:51.777128 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926d92b8-a1a0-4009-aa38-f1526cd8991b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:51.779498 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926d92b8-a1a0-4009-aa38-f1526cd8991b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:51.779498 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926d92b8-a1a0-4009-aa38-f1526cd8991b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:51.779498 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926d92b8-a1a0-4009-aa38-f1526cd8991b None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:51.779498 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926d92b8-a1a0-4009-aa38-f1526cd8991b None admin] RBAC: Adding query filter params (name=service) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:51.779801 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926d92b8-a1a0-4009-aa38-f1526cd8991b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:51.785229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926d92b8-a1a0-4009-aa38-f1526cd8991b None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:51.797099 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 392/780] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 18 00:43:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:51.797872 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926d92b8-a1a0-4009-aa38-f1526cd8991b None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:51.847201 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6666416a-d314-47bd-aba5-0948557a7ad7 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:51.903631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77458cf0-132b-4dd1-b08a-df20d67cf60a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:51.904244 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77458cf0-132b-4dd1-b08a-df20d67cf60a None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:51.904520 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77458cf0-132b-4dd1-b08a-df20d67cf60a None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:51.904797 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77458cf0-132b-4dd1-b08a-df20d67cf60a None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:51.905570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77458cf0-132b-4dd1-b08a-df20d67cf60a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:51.905865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77458cf0-132b-4dd1-b08a-df20d67cf60a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:51.907496 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77458cf0-132b-4dd1-b08a-df20d67cf60a None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:51.929088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 389/781] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:51.930573 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e875bfa5-d899-4232-b0f9-3968a8e0d4a9 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:51.931170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e875bfa5-d899-4232-b0f9-3968a8e0d4a9 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:51.931443 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e875bfa5-d899-4232-b0f9-3968a8e0d4a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:51.944024 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926d92b8-a1a0-4009-aa38-f1526cd8991b None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:51.952500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-e875bfa5-d899-4232-b0f9-3968a8e0d4a9 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:51.953727 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-e875bfa5-d899-4232-b0f9-3968a8e0d4a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:51.989295 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9d3b2a-70e6-4424-a14c-4521b83ab19c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:51.990228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9d3b2a-70e6-4424-a14c-4521b83ab19c None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:51.990660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9d3b2a-70e6-4424-a14c-4521b83ab19c None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:51.991063 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9d3b2a-70e6-4424-a14c-4521b83ab19c None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:51.992132 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9d3b2a-70e6-4424-a14c-4521b83ab19c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:51.994979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9d3b2a-70e6-4424-a14c-4521b83ab19c None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:52.014504 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 393/782] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Sep 18 00:43:51 2022] GET /identity/v3/services => generated 772 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:52.014504 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9d3b2a-70e6-4424-a14c-4521b83ab19c None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:52.039645 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:52.053649 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 390/783] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:51 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:43:52.054600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e875bfa5-d899-4232-b0f9-3968a8e0d4a9 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:52.122882 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:52.261735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9d3b2a-70e6-4424-a14c-4521b83ab19c None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:52.298061 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea460e66-2e02-4951-83f0-4439c237efdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:52.298805 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea460e66-2e02-4951-83f0-4439c237efdb None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:52.299156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea460e66-2e02-4951-83f0-4439c237efdb None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:52.299584 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea460e66-2e02-4951-83f0-4439c237efdb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:52.312672 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea460e66-2e02-4951-83f0-4439c237efdb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:52.314503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea460e66-2e02-4951-83f0-4439c237efdb None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:52.331398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-ea460e66-2e02-4951-83f0-4439c237efdb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:52.331398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:52.331398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:52.331398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:52.331398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:52.331398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:52.331398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:52.331398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:52.331398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:52.331398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:52.331398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:52.331398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:52.331398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:52.331398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:52.331398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:52.331398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:52.333183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:52.333183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:52.333183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:52.333183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:52.333183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:52.333183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:52.333183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:52.333183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:52.333183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:52.333183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:52.333183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:52.333183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:52.333183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:52.333183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:52.333183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:52.333183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:52.333183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:52.334918 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:52.334918 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:52.334918 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:52.334918 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 394/784] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 328 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:52.343795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea460e66-2e02-4951-83f0-4439c237efdb None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:52.354148 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e875bfa5-d899-4232-b0f9-3968a8e0d4a9 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:52.444838 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858dd2ae-26f0-47bc-9407-3f728e133842 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:52.447550 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858dd2ae-26f0-47bc-9407-3f728e133842 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:52.448494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858dd2ae-26f0-47bc-9407-3f728e133842 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:52.449582 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858dd2ae-26f0-47bc-9407-3f728e133842 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:52.467963 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858dd2ae-26f0-47bc-9407-3f728e133842 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:52.470585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858dd2ae-26f0-47bc-9407-3f728e133842 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:52.490873 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:52.495765 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea460e66-2e02-4951-83f0-4439c237efdb None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:52.497593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-858dd2ae-26f0-47bc-9407-3f728e133842 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:52.497593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:52.497593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:52.497593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:52.497593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:52.497593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:52.497593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:52.497593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:52.497593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:52.497593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:52.497593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:52.497593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:52.497593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:43:52.497593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:43:52.497593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:43:52.498522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:43:52.498522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:52.498522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:52.498522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:52.498522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:52.498522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:52.498522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:52.498522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:52.498522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:52.498522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:52.498522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:52.498522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:52.498522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:52.498522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:52.498522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:52.498522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:52.498522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:52.499425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:43:52.499425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:43:52.499425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:43:52.499425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:43:52.499425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:43:52.499425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:43:52.499425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:52.499425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:52.500527 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 391/785] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 445 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:52.501861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858dd2ae-26f0-47bc-9407-3f728e133842 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:52.554427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74dbbac2-296f-4164-a5a3-0c8877cfde75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:52.555100 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dbbac2-296f-4164-a5a3-0c8877cfde75 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:52.555382 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dbbac2-296f-4164-a5a3-0c8877cfde75 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:52.555693 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dbbac2-296f-4164-a5a3-0c8877cfde75 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:52.556414 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74dbbac2-296f-4164-a5a3-0c8877cfde75 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:52.557254 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74dbbac2-296f-4164-a5a3-0c8877cfde75 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:52.558270 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74dbbac2-296f-4164-a5a3-0c8877cfde75 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:52.574660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 395/786] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:52.574660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74dbbac2-296f-4164-a5a3-0c8877cfde75 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:52.586864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:52.647402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858dd2ae-26f0-47bc-9407-3f728e133842 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:52.697639 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3996ef-b786-4bc7-99ec-51eb2a9a0589 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:52.697992 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74dbbac2-296f-4164-a5a3-0c8877cfde75 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:52.699500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3996ef-b786-4bc7-99ec-51eb2a9a0589 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:52.700108 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3996ef-b786-4bc7-99ec-51eb2a9a0589 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:52.700663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3996ef-b786-4bc7-99ec-51eb2a9a0589 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:52.717927 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3996ef-b786-4bc7-99ec-51eb2a9a0589 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:52.723898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-df3996ef-b786-4bc7-99ec-51eb2a9a0589 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:43:52.723898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:52.723898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:52.723898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:52.723898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:52.723898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:52.723898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:52.723898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:52.723898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:52.723898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:52.723898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:52.723898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:52.723898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:52.723898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:52.723898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:52.723898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:52.725000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:52.725000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:52.725000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:52.725000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:52.725000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:52.725000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:52.725000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:52.725000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:52.725000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:52.725000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:52.725000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:52.725000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:52.725000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:52.725000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:52.725000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:52.725000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:52.725844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:52.725844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:52.725844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:52.725844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:52.725844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:52.725844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:52.725844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:52.725844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:52.725844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:52.725844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:52.725844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:43:52.725844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:52.725844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 392/787] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:43:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:52.726597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3996ef-b786-4bc7-99ec-51eb2a9a0589 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:52.745964 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56eb8d47-fa16-40d9-aeb5-9510c650c65b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:52.746927 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56eb8d47-fa16-40d9-aeb5-9510c650c65b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:52.747400 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56eb8d47-fa16-40d9-aeb5-9510c650c65b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:52.747832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56eb8d47-fa16-40d9-aeb5-9510c650c65b None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:52.748741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56eb8d47-fa16-40d9-aeb5-9510c650c65b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:52.749213 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56eb8d47-fa16-40d9-aeb5-9510c650c65b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:52.750928 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56eb8d47-fa16-40d9-aeb5-9510c650c65b None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:52.765156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 396/788] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:43:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:52.766428 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56eb8d47-fa16-40d9-aeb5-9510c650c65b None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:52.831191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:52.853031 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3996ef-b786-4bc7-99ec-51eb2a9a0589 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:52.881919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:52.896514 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56eb8d47-fa16-40d9-aeb5-9510c650c65b None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:52.899101 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01474be4-01ef-4565-a69d-f96150196886 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:52.900780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01474be4-01ef-4565-a69d-f96150196886 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:52.900780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01474be4-01ef-4565-a69d-f96150196886 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:52.901167 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01474be4-01ef-4565-a69d-f96150196886 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:52.920951 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01474be4-01ef-4565-a69d-f96150196886 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:52.923678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01474be4-01ef-4565-a69d-f96150196886 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:52.941996 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ef6aef-9321-4f80-8b70-1c021a110af8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:52.943014 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ef6aef-9321-4f80-8b70-1c021a110af8 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:52.943547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ef6aef-9321-4f80-8b70-1c021a110af8 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:52.944133 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ef6aef-9321-4f80-8b70-1c021a110af8 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:52.945023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ef6aef-9321-4f80-8b70-1c021a110af8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:52.945538 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ef6aef-9321-4f80-8b70-1c021a110af8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:52.946681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-01474be4-01ef-4565-a69d-f96150196886 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 18 00:43:52.946681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:52.946681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:52.946681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:52.946681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:52.946681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:52.946681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:52.946681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:52.946681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:52.946681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:52.946681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:52.946681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:52.946681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:43:52.946681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:43:52.946681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:43:52.946681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:43:52.947596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:52.947596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:52.947596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:43:52.947596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:52.947596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:43:52.947596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:52.947596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:52.947596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:52.947596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:52.947596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:52.947596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:52.947596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:52.947596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:52.947596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:52.947596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:52.947596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:52.947596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:52.947596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:52.948632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:52.948632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:52.948632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:43:52.948632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:43:52.948632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:43:52.948632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:43:52.948632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:43:52.948632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:43:52.948632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 18 00:43:52.948632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:52.949198 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 393/789] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Sep 18 00:43:52 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:52.950579 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ef6aef-9321-4f80-8b70-1c021a110af8 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:52.950732 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01474be4-01ef-4565-a69d-f96150196886 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:52.962309 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 397/790] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:43:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:52.963631 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ef6aef-9321-4f80-8b70-1c021a110af8 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:53.091964 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:53.119274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ef6aef-9321-4f80-8b70-1c021a110af8 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:53.145426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01474be4-01ef-4565-a69d-f96150196886 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:53.189040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61fdd4a-f293-457a-960e-b1fad2a8d438 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:53.189623 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61fdd4a-f293-457a-960e-b1fad2a8d438 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:53.190062 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61fdd4a-f293-457a-960e-b1fad2a8d438 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:53.190214 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61fdd4a-f293-457a-960e-b1fad2a8d438 None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:53.191136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61fdd4a-f293-457a-960e-b1fad2a8d438 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:53.191667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61fdd4a-f293-457a-960e-b1fad2a8d438 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:53.194443 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61fdd4a-f293-457a-960e-b1fad2a8d438 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:53.212495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564b16ba-de2c-4933-a2ba-adab0011f5c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:53.213177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564b16ba-de2c-4933-a2ba-adab0011f5c2 None admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:53.213453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564b16ba-de2c-4933-a2ba-adab0011f5c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:53.214193 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564b16ba-de2c-4933-a2ba-adab0011f5c2 None admin] PATH_INFO: `/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/a59d73fd45a54651930d52300704fdb0/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:53.216026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564b16ba-de2c-4933-a2ba-adab0011f5c2 None admin] RBAC: Authorizing `identity:create_grant(project_id=c520a8da3dc54adba4f7cc04ef42e367, user_id=a59d73fd45a54651930d52300704fdb0, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:53.217596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564b16ba-de2c-4933-a2ba-adab0011f5c2 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:53.232637 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 398/791] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Sep 18 00:43:52 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:53.234636 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61fdd4a-f293-457a-960e-b1fad2a8d438 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:53.240887 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 394/792] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:43:52 2022] PUT /identity/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/a59d73fd45a54651930d52300704fdb0/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:43:53.243065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564b16ba-de2c-4933-a2ba-adab0011f5c2 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:53.417500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:53.493698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61fdd4a-f293-457a-960e-b1fad2a8d438 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:53.520202 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564b16ba-de2c-4933-a2ba-adab0011f5c2 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:53.538367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba119da-8e5f-45b4-a12d-1bea4d277d7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:53.539060 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba119da-8e5f-45b4-a12d-1bea4d277d7e None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:53.539341 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba119da-8e5f-45b4-a12d-1bea4d277d7e None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:53.539613 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba119da-8e5f-45b4-a12d-1bea4d277d7e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:53.548219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:53.554784 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba119da-8e5f-45b4-a12d-1bea4d277d7e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:53.556392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba119da-8e5f-45b4-a12d-1bea4d277d7e None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:53.564173 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9128fa06-999f-4d1b-801b-b0dcbfebb02e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:53.564892 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9128fa06-999f-4d1b-801b-b0dcbfebb02e None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:53.565173 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9128fa06-999f-4d1b-801b-b0dcbfebb02e None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:53.565537 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9128fa06-999f-4d1b-801b-b0dcbfebb02e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:53.574265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-dba119da-8e5f-45b4-a12d-1bea4d277d7e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:53.574265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:53.574265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:53.574265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:53.574265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:53.574265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:53.574265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:53.574265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:53.574265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:53.574265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:53.574265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:53.574265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:53.574265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:53.574265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:53.574265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:53.574265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:53.575290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:53.575290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:53.575290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:53.575290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:53.575290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:53.575290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:53.575290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:53.575290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:53.575290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:53.575290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:53.575290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:53.575290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:53.575290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:53.575290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:53.575290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:53.575290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:53.575290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:53.576197 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:53.576197 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:53.576197 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:53.576197 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 399/793] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 342 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:53.582237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9128fa06-999f-4d1b-801b-b0dcbfebb02e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:53.582506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba119da-8e5f-45b4-a12d-1bea4d277d7e None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:53.583772 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9128fa06-999f-4d1b-801b-b0dcbfebb02e None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:53.602337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-9128fa06-999f-4d1b-801b-b0dcbfebb02e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:53.602337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:53.602337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:53.602337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:53.602337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:53.602337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:53.602337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:53.602337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:53.602337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:53.602337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:53.602337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:53.602337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:53.602337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:53.602337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:53.602337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:53.602337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:53.604720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:53.604720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:53.604720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:53.604720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:53.604720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:53.604720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:53.604720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:53.604720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:53.604720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:53.604720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:53.604720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:53.604720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:53.604720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:53.604720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:53.604720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:53.604720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:53.604720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:53.606913 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:53.606913 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:53.606913 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:53.610750 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 395/794] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 363 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:53.611887 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9128fa06-999f-4d1b-801b-b0dcbfebb02e None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:53.729957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:53.746800 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba119da-8e5f-45b4-a12d-1bea4d277d7e None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:53.757803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9128fa06-999f-4d1b-801b-b0dcbfebb02e None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:53.795734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba129af-2a1f-4b28-b69e-71b604ce44ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:53.797152 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba129af-2a1f-4b28-b69e-71b604ce44ff None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:53.797434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba129af-2a1f-4b28-b69e-71b604ce44ff None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:53.797708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba129af-2a1f-4b28-b69e-71b604ce44ff None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:53.798376 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba129af-2a1f-4b28-b69e-71b604ce44ff None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:53.798706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba129af-2a1f-4b28-b69e-71b604ce44ff None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:53.800110 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba129af-2a1f-4b28-b69e-71b604ce44ff None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:53.816319 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07af5c7c-fc04-491a-a9f8-dfecfd360f4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:53.816945 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07af5c7c-fc04-491a-a9f8-dfecfd360f4d None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:53.817223 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07af5c7c-fc04-491a-a9f8-dfecfd360f4d None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:53.817495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07af5c7c-fc04-491a-a9f8-dfecfd360f4d None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:53.818159 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07af5c7c-fc04-491a-a9f8-dfecfd360f4d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:53.818487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07af5c7c-fc04-491a-a9f8-dfecfd360f4d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:53.819868 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07af5c7c-fc04-491a-a9f8-dfecfd360f4d None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:53.833811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 400/795] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:43:53.842150 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 396/796] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:43:53.844928 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba129af-2a1f-4b28-b69e-71b604ce44ff None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:53.855120 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07af5c7c-fc04-491a-a9f8-dfecfd360f4d None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:53.914028 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:54.022592 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba129af-2a1f-4b28-b69e-71b604ce44ff None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:54.027965 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07af5c7c-fc04-491a-a9f8-dfecfd360f4d None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:54.070090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16619f7b-9f8d-4782-b47a-b29d2a488ddf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:54.071197 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16619f7b-9f8d-4782-b47a-b29d2a488ddf None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:54.071704 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16619f7b-9f8d-4782-b47a-b29d2a488ddf None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:54.072177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16619f7b-9f8d-4782-b47a-b29d2a488ddf None admin] PATH_INFO: `/v3/projects/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:54.090961 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16619f7b-9f8d-4782-b47a-b29d2a488ddf None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:54.094122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16619f7b-9f8d-4782-b47a-b29d2a488ddf None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:54.099101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5928a7e4-4d52-4954-8fe2-ceb674835b3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:54.100074 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5928a7e4-4d52-4954-8fe2-ceb674835b3b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:54.100560 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5928a7e4-4d52-4954-8fe2-ceb674835b3b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:54.101059 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5928a7e4-4d52-4954-8fe2-ceb674835b3b None admin] PATH_INFO: `/v3/users/swift` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:54.116652 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5928a7e4-4d52-4954-8fe2-ceb674835b3b None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:54.118166 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5928a7e4-4d52-4954-8fe2-ceb674835b3b None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:54.119738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-16619f7b-9f8d-4782-b47a-b29d2a488ddf None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:54.119738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:54.119738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:54.119738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:54.119738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:54.119738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:54.119738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:54.119738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:54.119738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:54.119738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:54.119738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:54.119738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:54.119738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:43:54.119738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:43:54.119738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:43:54.121206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:43:54.121206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:54.121206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:54.121206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:54.121206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:54.121206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:54.121206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:54.121206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:54.121206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:54.121206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:54.121206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:54.121206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:54.121206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:54.121206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:54.121206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:54.121206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:54.121206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:54.122080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:43:54.122080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:43:54.122080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:43:54.122080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:43:54.122080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:43:54.122080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:43:54.122080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:54.122080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:54.124638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 397/797] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:54.132529 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16619f7b-9f8d-4782-b47a-b29d2a488ddf None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:54.151693 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-5928a7e4-4d52-4954-8fe2-ceb674835b3b None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 18 00:43:54.151693 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:54.151693 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:54.151693 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:54.151693 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:54.151693 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:54.151693 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:54.151693 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:54.151693 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:54.151693 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:54.151693 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:54.151693 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:54.151693 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:43:54.151693 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:43:54.151693 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:43:54.151693 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:43:54.152706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:54.152706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:54.152706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:43:54.152706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:54.152706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:43:54.152706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:54.152706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:54.152706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:54.152706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:54.152706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:54.152706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:54.152706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:54.152706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:54.152706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:54.152706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:54.152706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:54.152706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:54.152706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:54.153646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:54.153646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:54.153646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:43:54.153646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:43:54.153646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:43:54.153646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:43:54.153646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:43:54.153646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:43:54.153646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 18 00:43:54.153646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:54.155285 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:54.159102 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 401/798] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:43:53 2022] GET /identity/v3/users/swift => generated 83 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:54.169821 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5928a7e4-4d52-4954-8fe2-ceb674835b3b None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:54.304212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16619f7b-9f8d-4782-b47a-b29d2a488ddf None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:54.402413 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d4f342-1827-4582-aec4-56af04209e66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:54.403486 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d4f342-1827-4582-aec4-56af04209e66 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:54.403999 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d4f342-1827-4582-aec4-56af04209e66 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:54.404485 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d4f342-1827-4582-aec4-56af04209e66 None admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:54.405459 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d4f342-1827-4582-aec4-56af04209e66 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:54.405978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d4f342-1827-4582-aec4-56af04209e66 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:54.407710 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d4f342-1827-4582-aec4-56af04209e66 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:54.439273 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 398/799] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:43:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:54.440751 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c263788a-5615-4455-a1fe-ea5e8e8f3188 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:54.441036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c263788a-5615-4455-a1fe-ea5e8e8f3188 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:54.441310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c263788a-5615-4455-a1fe-ea5e8e8f3188 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:54.446763 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 399/800] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:54.448152 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c263788a-5615-4455-a1fe-ea5e8e8f3188 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:54.456895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5928a7e4-4d52-4954-8fe2-ceb674835b3b None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:54.479808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:54.530058 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19564be8-49b2-4547-a446-fa8bd99177af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:54.531422 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19564be8-49b2-4547-a446-fa8bd99177af None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:54.532018 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19564be8-49b2-4547-a446-fa8bd99177af None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:54.532594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19564be8-49b2-4547-a446-fa8bd99177af None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:54.534223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19564be8-49b2-4547-a446-fa8bd99177af None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:54.535356 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19564be8-49b2-4547-a446-fa8bd99177af None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:54.538253 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19564be8-49b2-4547-a446-fa8bd99177af None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:54.539378 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:54.580888 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 402/801] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Sep 18 00:43:54 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 412 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:43:54.583257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd0f3bb-32fc-454c-aca3-51db1f603190 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:54.583651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd0f3bb-32fc-454c-aca3-51db1f603190 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:54.583912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c263788a-5615-4455-a1fe-ea5e8e8f3188 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:54.584252 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd0f3bb-32fc-454c-aca3-51db1f603190 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:54.617662 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-2dd0f3bb-32fc-454c-aca3-51db1f603190 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:54.621083 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-2dd0f3bb-32fc-454c-aca3-51db1f603190 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:54.625978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5566e9-e0e9-4489-b2da-1202e62dfef8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:54.626997 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5566e9-e0e9-4489-b2da-1202e62dfef8 None admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:54.627472 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5566e9-e0e9-4489-b2da-1202e62dfef8 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:54.629025 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5566e9-e0e9-4489-b2da-1202e62dfef8 None admin] PATH_INFO: `/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/3e31dce69c344c9aa4a55f31b9c735df/roles/81523e544c7e4993af4d1e06ce3754bc` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:54.648493 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5566e9-e0e9-4489-b2da-1202e62dfef8 None admin] RBAC: Authorizing `identity:create_grant(project_id=c520a8da3dc54adba4f7cc04ef42e367, user_id=3e31dce69c344c9aa4a55f31b9c735df, role_id=81523e544c7e4993af4d1e06ce3754bc)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:54.650214 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5566e9-e0e9-4489-b2da-1202e62dfef8 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:54.671241 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 400/802] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:43:54 2022] PUT /identity/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/3e31dce69c344c9aa4a55f31b9c735df/roles/81523e544c7e4993af4d1e06ce3754bc => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:43:54.672494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5566e9-e0e9-4489-b2da-1202e62dfef8 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:54.804054 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5566e9-e0e9-4489-b2da-1202e62dfef8 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:54.831023 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:54.849829 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e433fea-cb1f-4bf4-8dee-7b6231c1280c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:54.850550 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e433fea-cb1f-4bf4-8dee-7b6231c1280c None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:54.850844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e433fea-cb1f-4bf4-8dee-7b6231c1280c None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:54.851128 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e433fea-cb1f-4bf4-8dee-7b6231c1280c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:54.864530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e433fea-cb1f-4bf4-8dee-7b6231c1280c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:54.866174 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e433fea-cb1f-4bf4-8dee-7b6231c1280c None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:54.895365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-2e433fea-cb1f-4bf4-8dee-7b6231c1280c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:54.895365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:54.895365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:54.895365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:54.895365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:54.895365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:54.895365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:54.895365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:54.895365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:54.895365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:54.895365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:54.895365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:54.895365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:54.895365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:54.895365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:54.895365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:54.896487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:54.896487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:54.896487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:54.896487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:54.896487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:54.896487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:54.896487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:54.896487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:54.896487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:54.896487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:54.896487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:54.896487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:54.896487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:54.896487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:54.896487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:54.896487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:54.896487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:54.897540 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:54.897540 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:54.897540 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:54.897724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 401/803] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:54.900143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95756575-94b0-42b7-a6f8-45c4371f11a2 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:54.900523 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95756575-94b0-42b7-a6f8-45c4371f11a2 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:54.900935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95756575-94b0-42b7-a6f8-45c4371f11a2 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:54.903739 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 402/804] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:54 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:54.908718 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95756575-94b0-42b7-a6f8-45c4371f11a2 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:54.983536 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 403/805] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:54 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 402 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:43:54.985855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8287ea77-0809-40ae-9dc1-9acffcbbceb2 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:54.986314 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8287ea77-0809-40ae-9dc1-9acffcbbceb2 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:54.986813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8287ea77-0809-40ae-9dc1-9acffcbbceb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:55.017077 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-8287ea77-0809-40ae-9dc1-9acffcbbceb2 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:55.019206 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-8287ea77-0809-40ae-9dc1-9acffcbbceb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:55.141690 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95756575-94b0-42b7-a6f8-45c4371f11a2 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:55.156912 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 404/806] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:54 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:43:55.219551 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb51741f-4e8a-478e-940d-324beb798797 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:55.220599 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb51741f-4e8a-478e-940d-324beb798797 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:55.221079 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb51741f-4e8a-478e-940d-324beb798797 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:55.221574 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb51741f-4e8a-478e-940d-324beb798797 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:55.222755 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb51741f-4e8a-478e-940d-324beb798797 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:55.223289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb51741f-4e8a-478e-940d-324beb798797 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:55.225853 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb51741f-4e8a-478e-940d-324beb798797 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:55.249167 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 403/807] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 341 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:55.257800 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8287ea77-0809-40ae-9dc1-9acffcbbceb2 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:55.312398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:55.467740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8287ea77-0809-40ae-9dc1-9acffcbbceb2 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:55.516599 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646a5a47-e246-41e5-aeb5-3b49a7a6cc12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:55.517252 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646a5a47-e246-41e5-aeb5-3b49a7a6cc12 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:55.517532 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646a5a47-e246-41e5-aeb5-3b49a7a6cc12 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:55.517798 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646a5a47-e246-41e5-aeb5-3b49a7a6cc12 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:55.529407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-646a5a47-e246-41e5-aeb5-3b49a7a6cc12 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:55.530945 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-646a5a47-e246-41e5-aeb5-3b49a7a6cc12 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:55.545675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-646a5a47-e246-41e5-aeb5-3b49a7a6cc12 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:55.545675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:55.545675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:55.545675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:55.545675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:55.545675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:55.545675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:55.545675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:55.545675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:55.545675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:55.545675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:55.545675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:55.545675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:43:55.545675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:43:55.545675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:43:55.546660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:43:55.546660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:55.546660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:55.546660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:55.546660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:55.546660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:55.546660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:55.546660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:55.546660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:55.546660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:55.546660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:55.546660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:55.546660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:55.546660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:55.546660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:55.546660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:55.546660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:55.547562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:43:55.547562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:43:55.547562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:43:55.547562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:43:55.547562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:43:55.547562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:43:55.547562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:55.547562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:55.547562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 405/808] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:55.559845 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646a5a47-e246-41e5-aeb5-3b49a7a6cc12 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:55.560033 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb51741f-4e8a-478e-940d-324beb798797 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:55.626085 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:55.760464 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:55.777857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb51741f-4e8a-478e-940d-324beb798797 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:55.784100 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646a5a47-e246-41e5-aeb5-3b49a7a6cc12 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:55.817106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237a0f53-2647-4be7-83bd-3ed47b7fe51a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:55.818037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237a0f53-2647-4be7-83bd-3ed47b7fe51a None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:55.818516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237a0f53-2647-4be7-83bd-3ed47b7fe51a None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:55.818964 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237a0f53-2647-4be7-83bd-3ed47b7fe51a None admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:55.819840 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237a0f53-2647-4be7-83bd-3ed47b7fe51a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:55.820350 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237a0f53-2647-4be7-83bd-3ed47b7fe51a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:55.821981 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237a0f53-2647-4be7-83bd-3ed47b7fe51a None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:55.829136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ade430-efd9-41a0-8deb-bd388c91fc1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:55.830105 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ade430-efd9-41a0-8deb-bd388c91fc1c None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:55.830611 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ade430-efd9-41a0-8deb-bd388c91fc1c None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:55.831192 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ade430-efd9-41a0-8deb-bd388c91fc1c None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:55.831977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ade430-efd9-41a0-8deb-bd388c91fc1c None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:55.833554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ade430-efd9-41a0-8deb-bd388c91fc1c None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:55.844397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-c4ade430-efd9-41a0-8deb-bd388c91fc1c None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 18 00:43:55.844397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:55.844397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:43:55.844397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:43:55.844397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:43:55.844397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:43:55.844397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:43:55.844397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:43:55.844397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 18 00:43:55.844397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:55.844397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:43:55.844397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:55.844397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:55.844397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:55.844397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:55.844397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:55.844397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:55.845427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:55.845427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:55.845427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:55.845427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:55.845427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:55.845427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:55.845427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:43:55.845427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:43:55.845427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:43:55.845427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:43:55.845427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:55.845427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:55.845427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:55.845427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:55.845427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:55.845427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:55.846264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:55.846264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:55.846264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:55.846264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:55.846264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:55.846264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:55.846264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:55.846264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:55.846264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:55.846264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:55.846264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:43:55.846264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:43:55.846264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 18 00:43:55.846264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:55.848847 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 404/809] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:43:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:55.849219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 406/810] 10.222.0.42 () {56 vars in 1129 bytes} [Sun Sep 18 00:43:55 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:43:55.849853 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237a0f53-2647-4be7-83bd-3ed47b7fe51a None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:55.853450 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ade430-efd9-41a0-8deb-bd388c91fc1c None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:55.971551 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:55.988672 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237a0f53-2647-4be7-83bd-3ed47b7fe51a None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:55.997166 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ade430-efd9-41a0-8deb-bd388c91fc1c None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:56.032798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36755678-a497-45a0-97eb-447d13bbeafb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:56.034121 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36755678-a497-45a0-97eb-447d13bbeafb None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:56.034650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36755678-a497-45a0-97eb-447d13bbeafb None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:56.035129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36755678-a497-45a0-97eb-447d13bbeafb None admin] PATH_INFO: `/v3/services/network` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:56.035983 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36755678-a497-45a0-97eb-447d13bbeafb None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:56.037859 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36755678-a497-45a0-97eb-447d13bbeafb None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:56.050530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-36755678-a497-45a0-97eb-447d13bbeafb None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 18 00:43:56.050530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:56.050530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:43:56.050530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:43:56.050530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:43:56.050530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:43:56.050530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:43:56.050530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:43:56.050530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 18 00:43:56.050530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:56.050530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:43:56.050530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:56.050530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:56.050530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:56.050530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:56.050530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:56.050530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:56.051833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:56.051833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:56.051833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:56.051833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:56.051833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:56.051833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:56.051833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:43:56.051833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:43:56.051833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:43:56.051833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:43:56.051833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:56.051833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:56.051833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:56.051833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:56.051833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:56.051833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:56.052660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:56.052660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:56.052660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:56.052660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:56.052660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:56.052660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:56.052660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:56.052660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:56.052660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:56.052660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:56.052660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:43:56.052660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:43:56.052660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 18 00:43:56.052660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:56.055932 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 405/811] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:55 2022] GET /identity/v3/services/network => generated 88 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:56.057216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36755678-a497-45a0-97eb-447d13bbeafb None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:56.076692 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9627751-37d7-46d3-afd3-cb9699382d0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:56.078363 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9627751-37d7-46d3-afd3-cb9699382d0e None admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:56.079259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9627751-37d7-46d3-afd3-cb9699382d0e None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:56.079796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9627751-37d7-46d3-afd3-cb9699382d0e None admin] PATH_INFO: `/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/6230ed7ce0eb43e9b2b8530a28eb6299/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:56.083284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9627751-37d7-46d3-afd3-cb9699382d0e None admin] RBAC: Authorizing `identity:create_grant(project_id=c520a8da3dc54adba4f7cc04ef42e367, user_id=6230ed7ce0eb43e9b2b8530a28eb6299, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:56.086098 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9627751-37d7-46d3-afd3-cb9699382d0e None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:56.122488 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 407/812] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:43:55 2022] PUT /identity/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/6230ed7ce0eb43e9b2b8530a28eb6299/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:43:56.122488 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9627751-37d7-46d3-afd3-cb9699382d0e None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:56.152575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:56.215848 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36755678-a497-45a0-97eb-447d13bbeafb None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:56.258552 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe45e1d-d96f-4fa3-807e-8e2360be35ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:56.259193 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe45e1d-d96f-4fa3-807e-8e2360be35ac None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:56.259670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe45e1d-d96f-4fa3-807e-8e2360be35ac None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:56.259945 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe45e1d-d96f-4fa3-807e-8e2360be35ac None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:56.261688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe45e1d-d96f-4fa3-807e-8e2360be35ac None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:56.262283 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe45e1d-d96f-4fa3-807e-8e2360be35ac None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:56.263990 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe45e1d-d96f-4fa3-807e-8e2360be35ac None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:56.267680 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9627751-37d7-46d3-afd3-cb9699382d0e None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:56.270841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 406/813] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Sep 18 00:43:56 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:56.272315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c1c9f3-ea85-4909-98ff-17921f80e689 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:56.272679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c1c9f3-ea85-4909-98ff-17921f80e689 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:56.272895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c1c9f3-ea85-4909-98ff-17921f80e689 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:56.274546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 407/814] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:43:56.280405 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b374baa-0959-4073-86c0-500468db4e4e None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:56.280685 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b374baa-0959-4073-86c0-500468db4e4e None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:56.280970 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b374baa-0959-4073-86c0-500468db4e4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:56.301758 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-6b374baa-0959-4073-86c0-500468db4e4e None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:56.303130 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-6b374baa-0959-4073-86c0-500468db4e4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:56.316460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:56.319047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82822a8c-f07b-4b2a-8b56-3d4c2fe1e6ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:56.320214 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82822a8c-f07b-4b2a-8b56-3d4c2fe1e6ac None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:56.320713 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82822a8c-f07b-4b2a-8b56-3d4c2fe1e6ac None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:56.321183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82822a8c-f07b-4b2a-8b56-3d4c2fe1e6ac None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:56.322139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82822a8c-f07b-4b2a-8b56-3d4c2fe1e6ac None admin] RBAC: Adding query filter params (name=network) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:56.322628 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82822a8c-f07b-4b2a-8b56-3d4c2fe1e6ac None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:56.324991 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82822a8c-f07b-4b2a-8b56-3d4c2fe1e6ac None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:56.331252 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 408/815] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 18 00:43:56 2022] GET /identity/v3/services?name=network => generated 124 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:56.332590 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82822a8c-f07b-4b2a-8b56-3d4c2fe1e6ac None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:56.391928 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 408/816] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:56 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 18 00:43:56.393026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b374baa-0959-4073-86c0-500468db4e4e None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:56.448457 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:56.491563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82822a8c-f07b-4b2a-8b56-3d4c2fe1e6ac None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:56.542642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e2dce0-f9dd-4114-af84-1000e319e228 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:56.543278 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e2dce0-f9dd-4114-af84-1000e319e228 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:56.543531 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e2dce0-f9dd-4114-af84-1000e319e228 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:56.543809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e2dce0-f9dd-4114-af84-1000e319e228 None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:56.544521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e2dce0-f9dd-4114-af84-1000e319e228 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:56.544862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e2dce0-f9dd-4114-af84-1000e319e228 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:56.546705 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e2dce0-f9dd-4114-af84-1000e319e228 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:56.556159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 409/817] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Sep 18 00:43:56 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:56.564013 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e2dce0-f9dd-4114-af84-1000e319e228 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:56.565468 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b374baa-0959-4073-86c0-500468db4e4e None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:56.607372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3080088-9419-4e1a-9f88-c2d25bc47a45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:56.608376 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3080088-9419-4e1a-9f88-c2d25bc47a45 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:56.608888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3080088-9419-4e1a-9f88-c2d25bc47a45 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:56.609383 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3080088-9419-4e1a-9f88-c2d25bc47a45 None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:56.610293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3080088-9419-4e1a-9f88-c2d25bc47a45 None admin] RBAC: Adding query filter params (type=network) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:56.610857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3080088-9419-4e1a-9f88-c2d25bc47a45 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:56.612474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3080088-9419-4e1a-9f88-c2d25bc47a45 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:56.618733 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 409/818] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 18 00:43:56 2022] GET /identity/v3/services?type=network => generated 350 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:56.627746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3080088-9419-4e1a-9f88-c2d25bc47a45 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:56.658252 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:56.718227 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e2dce0-f9dd-4114-af84-1000e319e228 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:56.758473 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd5963c-4b26-498e-8f83-743deb2adc62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:56.759132 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd5963c-4b26-498e-8f83-743deb2adc62 None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:56.759417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd5963c-4b26-498e-8f83-743deb2adc62 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:56.759693 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd5963c-4b26-498e-8f83-743deb2adc62 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:56.761461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd5963c-4b26-498e-8f83-743deb2adc62 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:56.762925 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd5963c-4b26-498e-8f83-743deb2adc62 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:56.797348 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:56.821896 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 410/819] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:43:56 2022] POST /identity/v3/endpoints => generated 346 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:43:56.823761 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd5963c-4b26-498e-8f83-743deb2adc62 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:56.957018 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3080088-9419-4e1a-9f88-c2d25bc47a45 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:56.985992 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd5963c-4b26-498e-8f83-743deb2adc62 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:57.030263 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5254ca1d-1e7c-44da-af7c-7d2e9a99c7b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:57.031005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5254ca1d-1e7c-44da-af7c-7d2e9a99c7b7 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:57.031287 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5254ca1d-1e7c-44da-af7c-7d2e9a99c7b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:57.031565 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5254ca1d-1e7c-44da-af7c-7d2e9a99c7b7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:57.045596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5254ca1d-1e7c-44da-af7c-7d2e9a99c7b7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:57.052547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-5254ca1d-1e7c-44da-af7c-7d2e9a99c7b7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:43:57.052547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:57.052547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:57.052547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:57.052547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:57.052547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:57.052547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:57.052547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:57.052547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:57.052547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:57.052547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:57.052547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:57.052547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:57.052547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:57.052547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:57.052547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:57.053563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:57.053563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:57.053563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:57.053563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:57.053563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:57.053563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:57.053563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:57.053563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:57.053563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:57.053563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:57.053563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:57.053563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:57.053563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:57.053563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:57.053563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:57.053563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:57.054530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:57.054530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:57.054530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:57.054530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:57.054530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:57.054530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:57.054530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:57.054530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:57.054530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:57.054530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:57.054530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:43:57.054530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:57.054530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 411/820] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:43:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:57.064941 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5163eda8-16df-462c-9e51-596c75303bc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:57.066005 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5163eda8-16df-462c-9e51-596c75303bc9 None admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:57.066526 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5163eda8-16df-462c-9e51-596c75303bc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:57.067115 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5163eda8-16df-462c-9e51-596c75303bc9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:57.068099 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5163eda8-16df-462c-9e51-596c75303bc9 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:57.069915 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5163eda8-16df-462c-9e51-596c75303bc9 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:57.074938 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5254ca1d-1e7c-44da-af7c-7d2e9a99c7b7 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:57.105918 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 410/821] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:43:56 2022] POST /identity/v3/endpoints => generated 337 bytes in 478 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:43:57.145660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:57.210525 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5254ca1d-1e7c-44da-af7c-7d2e9a99c7b7 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:57.269790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d607be0-400e-4149-9573-78af8209c4bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:57.270437 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d607be0-400e-4149-9573-78af8209c4bc None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:57.270797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d607be0-400e-4149-9573-78af8209c4bc None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:57.271078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d607be0-400e-4149-9573-78af8209c4bc None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:57.271858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d607be0-400e-4149-9573-78af8209c4bc None admin] RBAC: Adding query filter params (name=admin) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:57.272165 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d607be0-400e-4149-9573-78af8209c4bc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:57.273605 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d607be0-400e-4149-9573-78af8209c4bc None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:57.285461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 412/822] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:43:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:43:57.291378 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5163eda8-16df-462c-9e51-596c75303bc9 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:57.372601 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:57.417706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5163eda8-16df-462c-9e51-596c75303bc9 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:57.458245 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96fd57f-bdd6-467a-a88d-c534b7a983ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:57.459264 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96fd57f-bdd6-467a-a88d-c534b7a983ae None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:57.459725 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96fd57f-bdd6-467a-a88d-c534b7a983ae None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:57.460201 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96fd57f-bdd6-467a-a88d-c534b7a983ae None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:57.478503 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96fd57f-bdd6-467a-a88d-c534b7a983ae None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:57.480865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96fd57f-bdd6-467a-a88d-c534b7a983ae None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:57.497779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-f96fd57f-bdd6-467a-a88d-c534b7a983ae None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:57.497779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:57.497779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:57.497779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:57.497779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:57.497779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:57.497779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:57.497779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:57.497779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:57.497779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:57.497779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:57.497779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:57.497779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:57.497779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:57.497779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:57.497779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:57.499036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:57.499036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:57.499036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:57.499036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:57.499036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:57.499036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:57.499036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:57.499036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:57.499036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:57.499036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:57.499036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:57.499036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:57.499036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:57.499036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:57.499036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:57.499036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:57.499036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:57.500131 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:57.500131 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:57.500131 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:57.503376 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 411/823] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:57.507590 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d607be0-400e-4149-9573-78af8209c4bc None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:57.642184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d607be0-400e-4149-9573-78af8209c4bc None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:57.658031 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:57.689250 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef1d934-06b3-4c52-be90-a245ea308a4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:57.689893 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef1d934-06b3-4c52-be90-a245ea308a4f None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:57.690175 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef1d934-06b3-4c52-be90-a245ea308a4f None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:57.690447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef1d934-06b3-4c52-be90-a245ea308a4f None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:57.691164 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef1d934-06b3-4c52-be90-a245ea308a4f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:57.691470 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef1d934-06b3-4c52-be90-a245ea308a4f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:57.692907 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef1d934-06b3-4c52-be90-a245ea308a4f None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:57.710928 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 413/824] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:43:57.712647 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96fd57f-bdd6-467a-a88d-c534b7a983ae None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:57.883062 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96fd57f-bdd6-467a-a88d-c534b7a983ae None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:57.919987 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:57.953693 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b0724b-e39b-4230-9d85-edbd33a329c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:57.955237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b0724b-e39b-4230-9d85-edbd33a329c4 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:57.955991 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b0724b-e39b-4230-9d85-edbd33a329c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:57.956727 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b0724b-e39b-4230-9d85-edbd33a329c4 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:57.972025 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b0724b-e39b-4230-9d85-edbd33a329c4 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:57.973779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b0724b-e39b-4230-9d85-edbd33a329c4 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:57.989415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-24b0724b-e39b-4230-9d85-edbd33a329c4 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 18 00:43:57.989415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:57.989415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:57.989415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:57.989415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:57.989415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:57.989415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:57.989415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:57.989415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:57.989415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:57.989415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:57.989415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:57.989415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:43:57.989415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:43:57.989415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:43:57.989415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:43:57.991172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:57.991172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:57.991172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:43:57.991172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:57.991172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:43:57.991172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:57.991172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:57.991172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:57.991172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:57.991172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:57.991172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:57.991172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:57.991172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:57.991172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:57.991172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:57.991172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:57.991172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:57.991172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:57.992907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:57.992907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:57.992907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:43:57.992907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:43:57.992907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:43:57.992907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:43:57.992907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:43:57.992907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:43:57.992907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 18 00:43:57.992907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:57.992907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 412/825] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Sep 18 00:43:57 2022] GET /identity/v3/users/nova => generated 82 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:57.997538 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef1d934-06b3-4c52-be90-a245ea308a4f None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:58.111135 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef1d934-06b3-4c52-be90-a245ea308a4f None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:58.147981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437c864b-69da-4ddd-be81-c4c00e2dcebf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:58.148597 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437c864b-69da-4ddd-be81-c4c00e2dcebf None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:58.148881 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437c864b-69da-4ddd-be81-c4c00e2dcebf None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:58.149143 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437c864b-69da-4ddd-be81-c4c00e2dcebf None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:58.149888 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-437c864b-69da-4ddd-be81-c4c00e2dcebf None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:58.150195 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-437c864b-69da-4ddd-be81-c4c00e2dcebf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:58.152227 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-437c864b-69da-4ddd-be81-c4c00e2dcebf None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:58.171765 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 414/826] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Sep 18 00:43:57 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:58.177529 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b0724b-e39b-4230-9d85-edbd33a329c4 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:58.190250 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392ffb78-a588-4c1e-a1ab-d18f4325a747 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:58.190741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392ffb78-a588-4c1e-a1ab-d18f4325a747 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:58.190980 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392ffb78-a588-4c1e-a1ab-d18f4325a747 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:58.192174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 415/827] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:43:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:43:58.197951 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2302b8-2deb-4a2d-a4e9-5dc76f2071c2 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:58.198901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2302b8-2deb-4a2d-a4e9-5dc76f2071c2 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:58.199142 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2302b8-2deb-4a2d-a4e9-5dc76f2071c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:58.217604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-ef2302b8-2deb-4a2d-a4e9-5dc76f2071c2 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:43:58.218928 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-ef2302b8-2deb-4a2d-a4e9-5dc76f2071c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:43:58.248911 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:58.306510 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b0724b-e39b-4230-9d85-edbd33a329c4 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:58.306979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 416/828] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:43:58 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:43:58.345900 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623cb651-0c6f-4677-8d48-ff544e6077a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:58.346632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623cb651-0c6f-4677-8d48-ff544e6077a3 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:58.346913 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623cb651-0c6f-4677-8d48-ff544e6077a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:58.347171 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623cb651-0c6f-4677-8d48-ff544e6077a3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:58.398439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:58.423740 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623cb651-0c6f-4677-8d48-ff544e6077a3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:58.425118 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623cb651-0c6f-4677-8d48-ff544e6077a3 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:58.550007 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 413/829] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:43:58 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 18 00:43:58.555671 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2302b8-2deb-4a2d-a4e9-5dc76f2071c2 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:58.606717 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623cb651-0c6f-4677-8d48-ff544e6077a3 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:58.686317 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2302b8-2deb-4a2d-a4e9-5dc76f2071c2 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:58.705416 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:58.728220 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00388227-63bf-4365-894f-58b17931e4d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:58.729152 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00388227-63bf-4365-894f-58b17931e4d3 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:58.729630 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00388227-63bf-4365-894f-58b17931e4d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:58.730042 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00388227-63bf-4365-894f-58b17931e4d3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:58.730854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623cb651-0c6f-4677-8d48-ff544e6077a3 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:58.742366 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00388227-63bf-4365-894f-58b17931e4d3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:58.744180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00388227-63bf-4365-894f-58b17931e4d3 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:58.761790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-00388227-63bf-4365-894f-58b17931e4d3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:58.761790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:58.761790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:58.761790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:58.761790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:58.761790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:58.761790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:58.761790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:58.761790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:58.761790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:58.761790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:58.761790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:58.761790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:58.761790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:58.761790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:58.761790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:58.762981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:58.762981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:58.762981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:58.762981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:58.762981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:58.762981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:58.762981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:58.762981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:58.762981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:58.762981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:58.762981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:58.762981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:58.762981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:58.762981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:58.762981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:58.762981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:58.762981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:58.763882 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:58.763882 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:58.763882 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:58.765889 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 417/830] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:58.771482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00388227-63bf-4365-894f-58b17931e4d3 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:58.778108 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3002e13a-3517-4265-8a23-c155a6b13c67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:58.780002 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3002e13a-3517-4265-8a23-c155a6b13c67 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:58.780497 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3002e13a-3517-4265-8a23-c155a6b13c67 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:58.780951 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3002e13a-3517-4265-8a23-c155a6b13c67 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:58.789773 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:58.798531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3002e13a-3517-4265-8a23-c155a6b13c67 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:43:58.805651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-3002e13a-3517-4265-8a23-c155a6b13c67 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:58.805651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:58.805651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:58.805651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:58.805651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:58.805651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:58.805651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:58.805651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:58.805651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:58.805651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:58.805651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:58.805651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:58.805651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:43:58.805651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:43:58.805651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:43:58.805651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:43:58.806680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:43:58.806680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:43:58.806680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:43:58.806680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:43:58.806680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:58.806680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:58.806680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:58.806680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:58.806680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:58.806680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:58.806680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:58.806680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:58.806680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:58.806680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:58.806680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:58.806680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:58.807556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:58.807556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:58.807556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:58.807556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:58.807556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:43:58.807556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:43:58.807556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:43:58.807556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:43:58.807556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:43:58.807556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:43:58.807556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:43:58.807556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:58.810836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 414/831] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:43:58 2022] GET /identity/v3/roles/service => generated 85 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:43:58.814077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3002e13a-3517-4265-8a23-c155a6b13c67 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:58.905712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00388227-63bf-4365-894f-58b17931e4d3 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:58.950611 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce3eedd-735d-4bce-96d2-460f14430a77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:58.951156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce3eedd-735d-4bce-96d2-460f14430a77 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:58.951438 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce3eedd-735d-4bce-96d2-460f14430a77 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:58.951718 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce3eedd-735d-4bce-96d2-460f14430a77 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:58.952376 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce3eedd-735d-4bce-96d2-460f14430a77 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:58.952636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3002e13a-3517-4265-8a23-c155a6b13c67 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:58.952745 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce3eedd-735d-4bce-96d2-460f14430a77 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:58.954081 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce3eedd-735d-4bce-96d2-460f14430a77 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:58.968065 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 418/832] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:58.974093 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce3eedd-735d-4bce-96d2-460f14430a77 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:59.001632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f657d39f-44e4-49a5-9f92-0329ad903b74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:59.002815 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f657d39f-44e4-49a5-9f92-0329ad903b74 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:59.003274 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f657d39f-44e4-49a5-9f92-0329ad903b74 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:59.004172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f657d39f-44e4-49a5-9f92-0329ad903b74 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:59.005309 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f657d39f-44e4-49a5-9f92-0329ad903b74 None admin] RBAC: Adding query filter params (name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:59.005769 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f657d39f-44e4-49a5-9f92-0329ad903b74 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:59.008242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f657d39f-44e4-49a5-9f92-0329ad903b74 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:59.016481 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:59.022799 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 415/833] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 18 00:43:58 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:43:59.029437 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f657d39f-44e4-49a5-9f92-0329ad903b74 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:59.065613 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:59.099938 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce3eedd-735d-4bce-96d2-460f14430a77 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:59.140526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd08a57f-3ada-4428-88b8-2425c76bef7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:59.141652 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd08a57f-3ada-4428-88b8-2425c76bef7d None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:59.142037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd08a57f-3ada-4428-88b8-2425c76bef7d None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:59.142417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd08a57f-3ada-4428-88b8-2425c76bef7d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:59.152766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd08a57f-3ada-4428-88b8-2425c76bef7d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:59.155448 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd08a57f-3ada-4428-88b8-2425c76bef7d None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:59.168524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f657d39f-44e4-49a5-9f92-0329ad903b74 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:59.172967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-dd08a57f-3ada-4428-88b8-2425c76bef7d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:59.172967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:59.172967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:59.172967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:59.172967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:59.172967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:59.172967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:59.172967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:59.172967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:59.172967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:59.172967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:59.172967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:59.172967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:43:59.172967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:43:59.172967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:43:59.173983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:43:59.173983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:59.173983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:59.173983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:59.173983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:59.173983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:59.173983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:59.173983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:59.173983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:59.173983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:59.173983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:59.173983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:59.173983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:59.173983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:59.173983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:59.173983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:59.173983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:59.174914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:43:59.174914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:43:59.174914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:43:59.174914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:43:59.174914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:43:59.174914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:43:59.174914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:43:59.174914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:43:59.180689 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 419/834] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:43:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:59.182116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd08a57f-3ada-4428-88b8-2425c76bef7d None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:59.217966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836e9ea8-2d5b-4c67-947e-dcc2f903dbf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:59.219720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836e9ea8-2d5b-4c67-947e-dcc2f903dbf0 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:59.220242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836e9ea8-2d5b-4c67-947e-dcc2f903dbf0 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:59.220972 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836e9ea8-2d5b-4c67-947e-dcc2f903dbf0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:59.236978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836e9ea8-2d5b-4c67-947e-dcc2f903dbf0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:59.239481 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836e9ea8-2d5b-4c67-947e-dcc2f903dbf0 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:59.260042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-836e9ea8-2d5b-4c67-947e-dcc2f903dbf0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:59.260042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:59.260042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:59.260042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:59.260042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:59.260042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:59.260042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:59.260042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:59.260042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:59.260042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:59.260042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:59.260042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:59.260042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:43:59.260042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:43:59.260042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:43:59.260042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:43:59.261835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:59.261835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:59.261835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:59.261835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:59.261835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:59.261835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:59.261835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:59.261835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:59.261835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:59.261835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:59.261835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:59.261835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:59.261835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:59.261835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:59.261835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:59.261835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:59.261835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:43:59.263191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:43:59.263191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:43:59.263191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:59.266546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 416/835] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:43:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:43:59.269152 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836e9ea8-2d5b-4c67-947e-dcc2f903dbf0 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:59.332263 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd08a57f-3ada-4428-88b8-2425c76bef7d None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:59.373960 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:59.379780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:59.385003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f3e8fd-3ed9-4e0d-81a9-6ad26a97c7ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:59.386882 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f3e8fd-3ed9-4e0d-81a9-6ad26a97c7ac None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:59.388059 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f3e8fd-3ed9-4e0d-81a9-6ad26a97c7ac None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:59.388532 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f3e8fd-3ed9-4e0d-81a9-6ad26a97c7ac None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:59.390277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f3e8fd-3ed9-4e0d-81a9-6ad26a97c7ac None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:59.390861 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f3e8fd-3ed9-4e0d-81a9-6ad26a97c7ac None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:59.393365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f3e8fd-3ed9-4e0d-81a9-6ad26a97c7ac None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:59.412259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 420/836] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:43:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:59.421423 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f3e8fd-3ed9-4e0d-81a9-6ad26a97c7ac None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:59.466089 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836e9ea8-2d5b-4c67-947e-dcc2f903dbf0 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:59.530624 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44adf676-b4cb-43ca-9349-b1c09dcbcff1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:59.531646 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44adf676-b4cb-43ca-9349-b1c09dcbcff1 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:59.532137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44adf676-b4cb-43ca-9349-b1c09dcbcff1 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:59.532619 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44adf676-b4cb-43ca-9349-b1c09dcbcff1 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:59.534374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44adf676-b4cb-43ca-9349-b1c09dcbcff1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:59.534708 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44adf676-b4cb-43ca-9349-b1c09dcbcff1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:59.537353 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44adf676-b4cb-43ca-9349-b1c09dcbcff1 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:59.569632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 417/837] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:43:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:59.574788 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44adf676-b4cb-43ca-9349-b1c09dcbcff1 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:59.586606 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f3e8fd-3ed9-4e0d-81a9-6ad26a97c7ac None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:59.638679 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8ac0f4-12fb-4978-8077-8a69a99fb751 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:59.639741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8ac0f4-12fb-4978-8077-8a69a99fb751 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:59.640222 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8ac0f4-12fb-4978-8077-8a69a99fb751 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:59.640694 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8ac0f4-12fb-4978-8077-8a69a99fb751 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:59.641681 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8ac0f4-12fb-4978-8077-8a69a99fb751 None admin] RBAC: Adding query filter params (role.id=1e04c9fea65244d28e1e59fc4c64f32f, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=a59d73fd45a54651930d52300704fdb0) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:43:59.642208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8ac0f4-12fb-4978-8077-8a69a99fb751 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:59.643990 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8ac0f4-12fb-4978-8077-8a69a99fb751 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:59.659382 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 421/838] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:43:59 2022] GET /identity/v3/role_assignments?user.id=a59d73fd45a54651930d52300704fdb0&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=1e04c9fea65244d28e1e59fc4c64f32f => generated 608 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:43:59.787659 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44adf676-b4cb-43ca-9349-b1c09dcbcff1 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:43:59.816607 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:43:59.861543 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02825aa-8167-4a48-b6ad-065386e875e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:43:59.862794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02825aa-8167-4a48-b6ad-065386e875e9 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:43:59.863273 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02825aa-8167-4a48-b6ad-065386e875e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:43:59.863738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02825aa-8167-4a48-b6ad-065386e875e9 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:43:59.879476 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02825aa-8167-4a48-b6ad-065386e875e9 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:43:59.882381 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02825aa-8167-4a48-b6ad-065386e875e9 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:43:59.907501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-a02825aa-8167-4a48-b6ad-065386e875e9 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 18 00:43:59.907501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:43:59.907501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:43:59.907501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:43:59.907501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:43:59.907501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:43:59.907501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:43:59.907501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:43:59.907501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:43:59.907501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:43:59.907501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:43:59.907501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:43:59.907501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:43:59.907501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:43:59.907501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:43:59.907501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:43:59.908516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:43:59.908516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:43:59.908516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:43:59.908516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:59.908516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:43:59.908516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:43:59.908516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:43:59.908516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:43:59.908516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:43:59.908516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:43:59.908516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:43:59.908516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:43:59.908516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:43:59.908516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:43:59.908516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:43:59.908516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:43:59.908516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:43:59.908516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:43:59.909435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:43:59.909435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:43:59.909435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:43:59.909435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:43:59.909435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:43:59.909435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:43:59.909435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:43:59.909435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:43:59.909435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 18 00:43:59.909435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:43:59.914637 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 418/839] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Sep 18 00:43:59 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 337 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:43:59.919616 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8ac0f4-12fb-4978-8077-8a69a99fb751 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:43:59.994803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:00.049748 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8ac0f4-12fb-4978-8077-8a69a99fb751 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:00.090155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5afe32-6fd6-45ff-a6dc-08a730092c22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:00.090964 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5afe32-6fd6-45ff-a6dc-08a730092c22 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:00.091257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5afe32-6fd6-45ff-a6dc-08a730092c22 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:00.091533 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5afe32-6fd6-45ff-a6dc-08a730092c22 None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:00.092432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5afe32-6fd6-45ff-a6dc-08a730092c22 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:00.092826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5afe32-6fd6-45ff-a6dc-08a730092c22 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:00.094205 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e93dc46-071b-46bf-8b2c-cba8d2e9abf3 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:00.094396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5afe32-6fd6-45ff-a6dc-08a730092c22 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:00.095002 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e93dc46-071b-46bf-8b2c-cba8d2e9abf3 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:00.095596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e93dc46-071b-46bf-8b2c-cba8d2e9abf3 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:00.102780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 419/840] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:00 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:00.108733 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cdde67-91f7-4bd7-8452-e91e72a5f68b None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:00.109309 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cdde67-91f7-4bd7-8452-e91e72a5f68b None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:00.109865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cdde67-91f7-4bd7-8452-e91e72a5f68b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:00.138648 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 422/841] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Sep 18 00:43:59 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:00.141153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5afe32-6fd6-45ff-a6dc-08a730092c22 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:00.143516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-b5cdde67-91f7-4bd7-8452-e91e72a5f68b None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:00.145469 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-b5cdde67-91f7-4bd7-8452-e91e72a5f68b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:00.270173 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 420/842] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:00 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:00.324379 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a22afb-4cf9-4e35-a47b-5c533b38d72b None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:00.324732 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a22afb-4cf9-4e35-a47b-5c533b38d72b None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:00.325204 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a22afb-4cf9-4e35-a47b-5c533b38d72b None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:00.330722 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 421/843] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:00 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:44:00.336143 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5afe32-6fd6-45ff-a6dc-08a730092c22 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:00.339963 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8f0fc0-c7b8-45f1-aaa4-30c1f69bbe0a None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:00.340652 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8f0fc0-c7b8-45f1-aaa4-30c1f69bbe0a None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:00.341289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8f0fc0-c7b8-45f1-aaa4-30c1f69bbe0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:00.359517 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:00.376845 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-8b8f0fc0-c7b8-45f1-aaa4-30c1f69bbe0a None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:00.379883 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8b8f0fc0-c7b8-45f1-aaa4-30c1f69bbe0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:00.396488 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad4a6ef-3668-4dd5-a0cc-6200d0b28bc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:00.397698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad4a6ef-3668-4dd5-a0cc-6200d0b28bc9 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:00.397911 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad4a6ef-3668-4dd5-a0cc-6200d0b28bc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:00.398360 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad4a6ef-3668-4dd5-a0cc-6200d0b28bc9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:00.514491 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 422/844] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:00 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:00.539875 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad4a6ef-3668-4dd5-a0cc-6200d0b28bc9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:00.542886 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad4a6ef-3668-4dd5-a0cc-6200d0b28bc9 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:00.671448 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8f0fc0-c7b8-45f1-aaa4-30c1f69bbe0a None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:00.704376 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:00.727155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 423/845] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:44:00 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 587 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 18 00:44:00.728244 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad4a6ef-3668-4dd5-a0cc-6200d0b28bc9 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:00.763110 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:00.849979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad4a6ef-3668-4dd5-a0cc-6200d0b28bc9 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:00.862296 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8f0fc0-c7b8-45f1-aaa4-30c1f69bbe0a None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:00.904422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa64b8f-719d-405b-88f5-d45f92b2f755 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:00.905232 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa64b8f-719d-405b-88f5-d45f92b2f755 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:00.905624 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa64b8f-719d-405b-88f5-d45f92b2f755 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:00.905990 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa64b8f-719d-405b-88f5-d45f92b2f755 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:00.920526 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa64b8f-719d-405b-88f5-d45f92b2f755 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:44:00.926144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-cfa64b8f-719d-405b-88f5-d45f92b2f755 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:44:00.926144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:00.926144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:00.926144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:00.926144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:00.926144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:00.926144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:00.926144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:00.926144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:00.926144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:00.926144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:00.926144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:00.926144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:44:00.926144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:44:00.926144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:44:00.926144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:44:00.927786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:44:00.927786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:44:00.927786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:44:00.927786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:44:00.927786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:00.927786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:00.927786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:00.927786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:00.927786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:00.927786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:00.927786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:00.927786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:00.927786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:00.927786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:00.927786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:00.927786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:00.929662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:00.929662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:00.929662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:00.929662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:00.929662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:44:00.929662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:44:00.929662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:44:00.929662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:44:00.929662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:44:00.929662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:44:00.929662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:44:00.929662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:00.936258 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ca26ad-bde5-4ff8-aa76-d7a399805674 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:00.937152 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 423/846] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:44:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:00.937415 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ca26ad-bde5-4ff8-aa76-d7a399805674 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:00.938527 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ca26ad-bde5-4ff8-aa76-d7a399805674 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:00.938696 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa64b8f-719d-405b-88f5-d45f92b2f755 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:00.938853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ca26ad-bde5-4ff8-aa76-d7a399805674 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:00.963330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ca26ad-bde5-4ff8-aa76-d7a399805674 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:00.966004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ca26ad-bde5-4ff8-aa76-d7a399805674 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:00.992943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-67ca26ad-bde5-4ff8-aa76-d7a399805674 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:00.992943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:00.992943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:00.992943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:00.992943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:00.992943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:00.992943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:00.992943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:00.992943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:00.992943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:00.992943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:00.992943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:00.992943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:44:00.992943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:44:00.992943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:44:00.992943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:44:00.995604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:00.995604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:00.995604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:00.995604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:00.995604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:00.995604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:00.995604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:00.995604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:00.995604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:00.995604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:00.995604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:00.995604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:00.995604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:00.995604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:00.995604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:00.995604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:00.995604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:44:00.998372 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:44:00.998372 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:00.998372 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:01.000580 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 424/847] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:01.006503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ca26ad-bde5-4ff8-aa76-d7a399805674 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:01.069725 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:01.087250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa64b8f-719d-405b-88f5-d45f92b2f755 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:01.117364 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:01.144862 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b5caf2-aab7-43af-b15e-7305d9b48468 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:01.145853 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b5caf2-aab7-43af-b15e-7305d9b48468 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:01.146326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b5caf2-aab7-43af-b15e-7305d9b48468 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:01.146826 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b5caf2-aab7-43af-b15e-7305d9b48468 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:01.148002 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b5caf2-aab7-43af-b15e-7305d9b48468 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:01.148529 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b5caf2-aab7-43af-b15e-7305d9b48468 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:01.151033 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b5caf2-aab7-43af-b15e-7305d9b48468 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:01.166513 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 424/848] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:44:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:01.172101 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b5caf2-aab7-43af-b15e-7305d9b48468 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:01.190409 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ca26ad-bde5-4ff8-aa76-d7a399805674 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:01.247052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d98575-35f8-4f4b-9451-421ea72d604b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:01.248052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d98575-35f8-4f4b-9451-421ea72d604b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:01.248536 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d98575-35f8-4f4b-9451-421ea72d604b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:01.249025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d98575-35f8-4f4b-9451-421ea72d604b None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:01.250148 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d98575-35f8-4f4b-9451-421ea72d604b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:01.250726 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d98575-35f8-4f4b-9451-421ea72d604b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:01.253322 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d98575-35f8-4f4b-9451-421ea72d604b None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:01.274545 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 425/849] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:44:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:01.276667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d98575-35f8-4f4b-9451-421ea72d604b None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:01.332999 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b5caf2-aab7-43af-b15e-7305d9b48468 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:01.375636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b39739d-d496-42b3-a8f8-54f49de2db27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:01.376593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b39739d-d496-42b3-a8f8-54f49de2db27 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:01.377018 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b39739d-d496-42b3-a8f8-54f49de2db27 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:01.377440 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b39739d-d496-42b3-a8f8-54f49de2db27 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:01.378395 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b39739d-d496-42b3-a8f8-54f49de2db27 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:01.380710 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b39739d-d496-42b3-a8f8-54f49de2db27 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:01.396711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-1b39739d-d496-42b3-a8f8-54f49de2db27 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 18 00:44:01.396711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:01.396711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:44:01.396711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:44:01.396711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:44:01.396711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:44:01.396711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:44:01.396711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:01.396711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 18 00:44:01.396711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:01.396711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:44:01.396711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:01.396711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:01.396711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:01.396711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:01.396711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:01.396711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:01.397798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:01.397798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:01.397798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:01.397798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:01.397798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:01.397798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:01.397798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:44:01.397798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:44:01.397798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:44:01.397798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:44:01.397798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:01.397798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:01.397798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:01.397798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:01.397798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:01.397798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:01.398675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:01.398675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:01.398675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:01.398675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:01.398675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:01.398675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:01.398675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:01.398675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:01.398675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:01.398675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:01.398675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:44:01.398675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:01.398675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 18 00:44:01.398675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:01.398675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 425/850] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 18 00:44:01 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:01.399826 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b39739d-d496-42b3-a8f8-54f49de2db27 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:01.434168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:01.459825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d98575-35f8-4f4b-9451-421ea72d604b None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:01.495171 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:01.507399 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f149e131-6f30-4740-997a-c55e91a906c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:01.508125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f149e131-6f30-4740-997a-c55e91a906c7 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:01.508329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f149e131-6f30-4740-997a-c55e91a906c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:01.508601 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f149e131-6f30-4740-997a-c55e91a906c7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:01.523653 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f149e131-6f30-4740-997a-c55e91a906c7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:01.527190 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f149e131-6f30-4740-997a-c55e91a906c7 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:01.554187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-f149e131-6f30-4740-997a-c55e91a906c7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:01.554187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:01.554187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:01.554187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:01.554187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:01.554187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:01.554187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:01.554187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:01.554187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:01.554187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:01.554187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:01.554187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:01.554187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:44:01.554187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:44:01.554187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:44:01.554187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:44:01.555420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:01.555420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:01.555420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:01.555420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:01.555420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:01.555420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:01.555420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:01.555420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:01.555420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:01.555420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:01.555420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:01.555420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:01.555420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:01.555420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:01.555420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:01.555420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:01.555420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:44:01.556321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:44:01.556321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:01.556321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:01.556321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 426/851] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:01.557381 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f149e131-6f30-4740-997a-c55e91a906c7 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:01.587643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b39739d-d496-42b3-a8f8-54f49de2db27 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:01.643413 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4315aa88-32be-4c09-9514-461520c219b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:01.644388 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4315aa88-32be-4c09-9514-461520c219b4 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:01.644830 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4315aa88-32be-4c09-9514-461520c219b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:01.645276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4315aa88-32be-4c09-9514-461520c219b4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:01.656774 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4315aa88-32be-4c09-9514-461520c219b4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:01.658792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4315aa88-32be-4c09-9514-461520c219b4 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:01.678691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-4315aa88-32be-4c09-9514-461520c219b4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:44:01.678691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:01.678691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:01.678691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:01.678691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:01.678691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:01.678691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:01.678691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:01.678691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:01.678691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:01.678691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:01.678691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:01.678691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:44:01.678691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:44:01.678691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:44:01.680360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:44:01.680360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:01.680360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:01.680360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:01.680360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:01.680360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:01.680360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:01.680360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:01.680360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:01.680360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:01.680360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:01.680360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:01.680360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:01.680360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:01.680360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:01.680360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:01.680360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:01.683030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:44:01.683030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:44:01.683030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:44:01.683030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:44:01.683030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:44:01.683030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:44:01.683030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:44:01.683030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:01.683030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 426/852] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:44:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:01.683030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4315aa88-32be-4c09-9514-461520c219b4 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:01.734123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f149e131-6f30-4740-997a-c55e91a906c7 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:01.792690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9655c793-ee9b-4907-b71b-acee13ddcfed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:01.793497 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9655c793-ee9b-4907-b71b-acee13ddcfed None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:01.793945 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9655c793-ee9b-4907-b71b-acee13ddcfed None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:01.794383 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9655c793-ee9b-4907-b71b-acee13ddcfed None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:01.795105 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9655c793-ee9b-4907-b71b-acee13ddcfed None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:01.795452 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9655c793-ee9b-4907-b71b-acee13ddcfed None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:01.797318 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9655c793-ee9b-4907-b71b-acee13ddcfed None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:01.810511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 427/853] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 18 00:44:01 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:01.810511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9655c793-ee9b-4907-b71b-acee13ddcfed None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:01.815290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:01.852667 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4315aa88-32be-4c09-9514-461520c219b4 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:01.890383 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:01.922428 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5a5e37-d453-4e73-bb1f-6c5c77a5cd0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:01.923735 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5a5e37-d453-4e73-bb1f-6c5c77a5cd0b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:01.923988 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5a5e37-d453-4e73-bb1f-6c5c77a5cd0b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:01.924441 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5a5e37-d453-4e73-bb1f-6c5c77a5cd0b None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:01.925657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b5a5e37-d453-4e73-bb1f-6c5c77a5cd0b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:01.926503 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b5a5e37-d453-4e73-bb1f-6c5c77a5cd0b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:01.929057 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b5a5e37-d453-4e73-bb1f-6c5c77a5cd0b None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:01.956887 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 427/854] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:44:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:01.956887 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5a5e37-d453-4e73-bb1f-6c5c77a5cd0b None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:01.977288 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9655c793-ee9b-4907-b71b-acee13ddcfed None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:02.027105 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744080d9-b010-42a9-9a35-f71ae6306736 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:02.027629 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744080d9-b010-42a9-9a35-f71ae6306736 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:02.027919 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744080d9-b010-42a9-9a35-f71ae6306736 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:02.028166 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744080d9-b010-42a9-9a35-f71ae6306736 None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:02.028885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744080d9-b010-42a9-9a35-f71ae6306736 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:02.029186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744080d9-b010-42a9-9a35-f71ae6306736 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:02.030781 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744080d9-b010-42a9-9a35-f71ae6306736 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:02.055617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 428/855] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:44:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:02.057148 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744080d9-b010-42a9-9a35-f71ae6306736 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:02.145751 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5a5e37-d453-4e73-bb1f-6c5c77a5cd0b None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:02.202381 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70eb8896-94ed-40b9-bc91-8430c83b6c18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:02.203441 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70eb8896-94ed-40b9-bc91-8430c83b6c18 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:02.203985 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70eb8896-94ed-40b9-bc91-8430c83b6c18 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:02.204465 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70eb8896-94ed-40b9-bc91-8430c83b6c18 None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:02.205652 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70eb8896-94ed-40b9-bc91-8430c83b6c18 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:02.206255 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70eb8896-94ed-40b9-bc91-8430c83b6c18 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:02.209014 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70eb8896-94ed-40b9-bc91-8430c83b6c18 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:02.217579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 428/856] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 18 00:44:01 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:02.219012 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70eb8896-94ed-40b9-bc91-8430c83b6c18 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:02.234631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:02.251339 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:02.258888 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744080d9-b010-42a9-9a35-f71ae6306736 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:02.345994 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81c6c75-51d2-46d2-a3f6-1ef499e5ccd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:02.347599 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81c6c75-51d2-46d2-a3f6-1ef499e5ccd3 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:02.349423 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81c6c75-51d2-46d2-a3f6-1ef499e5ccd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:02.350249 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81c6c75-51d2-46d2-a3f6-1ef499e5ccd3 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:02.369585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81c6c75-51d2-46d2-a3f6-1ef499e5ccd3 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:02.373516 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81c6c75-51d2-46d2-a3f6-1ef499e5ccd3 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:02.405511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-a81c6c75-51d2-46d2-a3f6-1ef499e5ccd3 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 18 00:44:02.405511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:02.405511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:02.405511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:02.405511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:02.405511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:02.405511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:02.405511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:02.405511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:02.405511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:02.405511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:02.405511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:02.405511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:44:02.405511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:44:02.405511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:44:02.405511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:44:02.407461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:02.407461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:02.407461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:44:02.407461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:44:02.407461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:44:02.407461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:44:02.407461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:02.407461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:02.407461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:02.407461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:02.407461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:02.407461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:02.407461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:02.407461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:02.407461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:02.407461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:02.407461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:02.407461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:02.415500 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:02.415500 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:02.415500 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:44:02.415500 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:44:02.415500 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:44:02.415500 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:44:02.415500 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:44:02.415500 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:44:02.415500 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 18 00:44:02.415500 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:02.415500 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 429/857] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:44:02 2022] GET /identity/v3/users/swift => generated 83 bytes in 353 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:02.422591 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81c6c75-51d2-46d2-a3f6-1ef499e5ccd3 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:02.460722 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70eb8896-94ed-40b9-bc91-8430c83b6c18 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:02.534854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d0b0e1-5d10-453e-bf70-4c097bfc796f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:02.535560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d0b0e1-5d10-453e-bf70-4c097bfc796f None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:02.535841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d0b0e1-5d10-453e-bf70-4c097bfc796f None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:02.536111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d0b0e1-5d10-453e-bf70-4c097bfc796f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:02.536873 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d0b0e1-5d10-453e-bf70-4c097bfc796f None admin] RBAC: Adding query filter params (role.id=81523e544c7e4993af4d1e06ce3754bc, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=3e31dce69c344c9aa4a55f31b9c735df) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:02.537181 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d0b0e1-5d10-453e-bf70-4c097bfc796f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:02.538714 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d0b0e1-5d10-453e-bf70-4c097bfc796f None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:02.549420 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 429/858] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:44:02 2022] GET /identity/v3/role_assignments?user.id=3e31dce69c344c9aa4a55f31b9c735df&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=81523e544c7e4993af4d1e06ce3754bc => generated 608 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:02.549420 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeba7e04-1199-43ec-8ef1-e603c8fa1ccd None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:02.549420 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeba7e04-1199-43ec-8ef1-e603c8fa1ccd None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:02.549420 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeba7e04-1199-43ec-8ef1-e603c8fa1ccd None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:02.550003 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 430/859] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:02.565028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1010ccb9-6b83-434b-8ce5-a2483a594251 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:02.565466 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1010ccb9-6b83-434b-8ce5-a2483a594251 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:02.565613 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1010ccb9-6b83-434b-8ce5-a2483a594251 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:02.590934 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-1010ccb9-6b83-434b-8ce5-a2483a594251 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:02.591559 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81c6c75-51d2-46d2-a3f6-1ef499e5ccd3 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:02.592538 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-1010ccb9-6b83-434b-8ce5-a2483a594251 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:02.659555 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d914fd0-1666-432b-9c6c-5b995e3fef90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:02.660146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d914fd0-1666-432b-9c6c-5b995e3fef90 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:02.660403 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d914fd0-1666-432b-9c6c-5b995e3fef90 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:02.660676 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d914fd0-1666-432b-9c6c-5b995e3fef90 None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:02.661457 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d914fd0-1666-432b-9c6c-5b995e3fef90 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:02.661765 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d914fd0-1666-432b-9c6c-5b995e3fef90 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:02.663350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d914fd0-1666-432b-9c6c-5b995e3fef90 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:02.686530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 430/860] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Sep 18 00:44:02 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:02.691724 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d914fd0-1666-432b-9c6c-5b995e3fef90 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:02.693303 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:02.710696 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 431/861] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:02 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 18 00:44:02.716740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:02.827980 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d914fd0-1666-432b-9c6c-5b995e3fef90 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:02.888680 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14067b7a-17fb-453f-82d8-5c22887f4848 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:02.889662 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14067b7a-17fb-453f-82d8-5c22887f4848 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:02.890105 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14067b7a-17fb-453f-82d8-5c22887f4848 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:02.890554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14067b7a-17fb-453f-82d8-5c22887f4848 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:02.991985 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1010ccb9-6b83-434b-8ce5-a2483a594251 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:03.019806 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14067b7a-17fb-453f-82d8-5c22887f4848 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:03.022835 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14067b7a-17fb-453f-82d8-5c22887f4848 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:03.057404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:03.107591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1010ccb9-6b83-434b-8ce5-a2483a594251 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:03.145751 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fda4dd9-9a10-467c-bc63-f1e90536545f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:03.146402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fda4dd9-9a10-467c-bc63-f1e90536545f None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:03.146724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fda4dd9-9a10-467c-bc63-f1e90536545f None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:03.146997 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fda4dd9-9a10-467c-bc63-f1e90536545f None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:03.147597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fda4dd9-9a10-467c-bc63-f1e90536545f None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:03.149063 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fda4dd9-9a10-467c-bc63-f1e90536545f None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:03.158942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-5fda4dd9-9a10-467c-bc63-f1e90536545f None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 18 00:44:03.158942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:03.158942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:44:03.158942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:44:03.158942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:44:03.158942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:44:03.158942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:44:03.158942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:03.158942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 18 00:44:03.158942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:03.158942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:44:03.158942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:03.158942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:03.158942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:03.158942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:03.158942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:03.161276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:03.161276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:03.161276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:03.161276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:03.161276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:03.161276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:03.161276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:03.161276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:44:03.161276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:44:03.161276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:44:03.161276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:44:03.161276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:03.161276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:03.161276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:03.161276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:03.161276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:03.161276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:03.163546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:03.163546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:03.163546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:03.163546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:03.163546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:03.163546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:03.163546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:03.163546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:03.163546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:03.163546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:03.163546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:44:03.163546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:03.163546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 18 00:44:03.163546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:03.165391 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 432/862] 10.222.0.42 () {56 vars in 1132 bytes} [Sun Sep 18 00:44:02 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:03.174514 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fda4dd9-9a10-467c-bc63-f1e90536545f None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:03.212771 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 431/863] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:44:02 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 521 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 18 00:44:03.221353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14067b7a-17fb-453f-82d8-5c22887f4848 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:03.239766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:03.347414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fda4dd9-9a10-467c-bc63-f1e90536545f None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:03.359882 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14067b7a-17fb-453f-82d8-5c22887f4848 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:03.386842 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:03.413031 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1519a2ba-877e-4f43-8606-19f810135d22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:03.414056 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1519a2ba-877e-4f43-8606-19f810135d22 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:03.414610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1519a2ba-877e-4f43-8606-19f810135d22 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:03.415097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1519a2ba-877e-4f43-8606-19f810135d22 None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:03.416741 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1519a2ba-877e-4f43-8606-19f810135d22 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:03.417405 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1519a2ba-877e-4f43-8606-19f810135d22 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:03.419970 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1519a2ba-877e-4f43-8606-19f810135d22 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:03.427820 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a673dcec-0ca6-40b0-8a20-3e71f89a097b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:03.429354 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 433/864] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Sep 18 00:44:03 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:03.429768 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a673dcec-0ca6-40b0-8a20-3e71f89a097b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:03.430417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a673dcec-0ca6-40b0-8a20-3e71f89a097b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:03.430947 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a673dcec-0ca6-40b0-8a20-3e71f89a097b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:03.435413 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1519a2ba-877e-4f43-8606-19f810135d22 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:03.455242 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a673dcec-0ca6-40b0-8a20-3e71f89a097b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:03.459026 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a673dcec-0ca6-40b0-8a20-3e71f89a097b None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:03.498289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-a673dcec-0ca6-40b0-8a20-3e71f89a097b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:03.498289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:03.498289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:03.498289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:03.498289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:03.498289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:03.498289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:03.498289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:03.498289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:03.498289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:03.498289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:03.498289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:03.498289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:44:03.498289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:44:03.498289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:44:03.498289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:44:03.500191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:03.500191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:03.500191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:03.500191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:03.500191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:03.500191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:03.500191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:03.500191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:03.500191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:03.500191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:03.500191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:03.500191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:03.500191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:03.500191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:03.500191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:03.500191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:03.500191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:44:03.501885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:44:03.501885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:03.501885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:03.506081 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 432/865] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:03.512277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a673dcec-0ca6-40b0-8a20-3e71f89a097b None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:03.663111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1519a2ba-877e-4f43-8606-19f810135d22 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:03.692803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:03.706053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ce3ccc-4d76-44c0-a86e-b547be4ea705 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:03.706626 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ce3ccc-4d76-44c0-a86e-b547be4ea705 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:03.706904 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ce3ccc-4d76-44c0-a86e-b547be4ea705 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:03.707167 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ce3ccc-4d76-44c0-a86e-b547be4ea705 None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:03.707821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ce3ccc-4d76-44c0-a86e-b547be4ea705 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:03.708126 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ce3ccc-4d76-44c0-a86e-b547be4ea705 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:03.709661 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ce3ccc-4d76-44c0-a86e-b547be4ea705 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:03.715843 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 434/866] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Sep 18 00:44:03 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:03.721354 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a673dcec-0ca6-40b0-8a20-3e71f89a097b None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:03.806683 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01b0b2f-c187-4e72-8912-7392d99d6fad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:03.808451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01b0b2f-c187-4e72-8912-7392d99d6fad None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:03.809276 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01b0b2f-c187-4e72-8912-7392d99d6fad None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:03.810038 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01b0b2f-c187-4e72-8912-7392d99d6fad None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:03.811610 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01b0b2f-c187-4e72-8912-7392d99d6fad None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:03.812621 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01b0b2f-c187-4e72-8912-7392d99d6fad None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:03.815757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01b0b2f-c187-4e72-8912-7392d99d6fad None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:03.849658 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 433/867] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:44:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 333 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:03.855865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ce3ccc-4d76-44c0-a86e-b547be4ea705 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:04.033878 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ce3ccc-4d76-44c0-a86e-b547be4ea705 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:04.072815 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba2bcf8-8b6d-40e2-8aab-28a602b11950 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:04.073997 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba2bcf8-8b6d-40e2-8aab-28a602b11950 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:04.074342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba2bcf8-8b6d-40e2-8aab-28a602b11950 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:04.074649 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba2bcf8-8b6d-40e2-8aab-28a602b11950 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:04.082315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba2bcf8-8b6d-40e2-8aab-28a602b11950 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:04.083813 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba2bcf8-8b6d-40e2-8aab-28a602b11950 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:04.096702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-aba2bcf8-8b6d-40e2-8aab-28a602b11950 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:44:04.096702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:04.096702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:04.096702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:04.096702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:04.096702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:04.096702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:04.096702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:04.096702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:04.096702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:04.096702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:04.096702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:04.096702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:44:04.096702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:44:04.096702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:44:04.098818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:44:04.098818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:04.098818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:04.098818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:04.098818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:04.098818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:04.098818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:04.098818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:04.098818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:04.098818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:04.098818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:04.098818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:04.098818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:04.098818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:04.098818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:04.098818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:04.098818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:04.100721 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:44:04.100721 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:44:04.100721 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:44:04.100721 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:44:04.100721 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:44:04.100721 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:44:04.100721 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:44:04.100721 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:04.100721 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 435/868] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:44:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:04.111696 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01b0b2f-c187-4e72-8912-7392d99d6fad None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:04.171182 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:04.336825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01b0b2f-c187-4e72-8912-7392d99d6fad None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:04.399160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9603a9-93c3-4795-b6b0-fd482f414978 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:04.400548 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9603a9-93c3-4795-b6b0-fd482f414978 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:04.401351 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9603a9-93c3-4795-b6b0-fd482f414978 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:04.402089 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9603a9-93c3-4795-b6b0-fd482f414978 None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:04.403703 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9603a9-93c3-4795-b6b0-fd482f414978 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:04.404504 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9603a9-93c3-4795-b6b0-fd482f414978 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:04.407478 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9603a9-93c3-4795-b6b0-fd482f414978 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:04.434596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 434/869] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:44:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 320 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:04.441214 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba2bcf8-8b6d-40e2-8aab-28a602b11950 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:04.537480 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:04.610407 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba2bcf8-8b6d-40e2-8aab-28a602b11950 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:04.647656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca1755c-99b1-4535-b924-f3c1a4dd7360 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:04.648248 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca1755c-99b1-4535-b924-f3c1a4dd7360 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:04.648533 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca1755c-99b1-4535-b924-f3c1a4dd7360 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:04.648806 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca1755c-99b1-4535-b924-f3c1a4dd7360 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:04.649553 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca1755c-99b1-4535-b924-f3c1a4dd7360 None admin] RBAC: Adding query filter params (role.id=1e04c9fea65244d28e1e59fc4c64f32f, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=6230ed7ce0eb43e9b2b8530a28eb6299) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:04.649858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca1755c-99b1-4535-b924-f3c1a4dd7360 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:04.651391 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca1755c-99b1-4535-b924-f3c1a4dd7360 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:04.658847 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 436/870] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:44:04 2022] GET /identity/v3/role_assignments?user.id=6230ed7ce0eb43e9b2b8530a28eb6299&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=1e04c9fea65244d28e1e59fc4c64f32f => generated 608 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:04.793937 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6c3aea-7b05-49b5-b7aa-902c34050994 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:04.795134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6c3aea-7b05-49b5-b7aa-902c34050994 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:04.795952 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6c3aea-7b05-49b5-b7aa-902c34050994 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:04.799579 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 435/871] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:04 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:04.805318 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c282d7ee-3051-4589-b7b0-7d2eb984c11c None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:04.805605 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c282d7ee-3051-4589-b7b0-7d2eb984c11c None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:04.805879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c282d7ee-3051-4589-b7b0-7d2eb984c11c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:04.827230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-c282d7ee-3051-4589-b7b0-7d2eb984c11c None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:04.828520 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c282d7ee-3051-4589-b7b0-7d2eb984c11c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:04.917823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 437/872] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:04 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:05.187544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6c3aea-7b05-49b5-b7aa-902c34050994 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:05.291411 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:05.304108 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6c3aea-7b05-49b5-b7aa-902c34050994 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:05.342676 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b1cd97-a5a7-406a-aee9-35b5ca01f760 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:05.343316 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b1cd97-a5a7-406a-aee9-35b5ca01f760 None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:05.343594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b1cd97-a5a7-406a-aee9-35b5ca01f760 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:05.343865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b1cd97-a5a7-406a-aee9-35b5ca01f760 None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:05.344783 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b1cd97-a5a7-406a-aee9-35b5ca01f760 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:05.346312 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b1cd97-a5a7-406a-aee9-35b5ca01f760 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:05.364932 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 436/873] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 18 00:44:05 2022] POST /identity/v3/services => generated 251 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:44:05.480833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f647d0-5280-4f48-9b23-05304a5c3ac7 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:05.481562 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f647d0-5280-4f48-9b23-05304a5c3ac7 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:05.481562 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f647d0-5280-4f48-9b23-05304a5c3ac7 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:05.483096 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 438/874] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:44:05.489209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2428aade-ccdf-4bef-abe5-12f32d878eb3 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:05.489544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2428aade-ccdf-4bef-abe5-12f32d878eb3 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:05.489849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2428aade-ccdf-4bef-abe5-12f32d878eb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:05.511171 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-2428aade-ccdf-4bef-abe5-12f32d878eb3 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:05.512553 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-2428aade-ccdf-4bef-abe5-12f32d878eb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:05.599305 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 437/875] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:05 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:05.894021 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f647d0-5280-4f48-9b23-05304a5c3ac7 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:05.950600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:06.072113 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f647d0-5280-4f48-9b23-05304a5c3ac7 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:06.144401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd18753-37bd-4372-9102-3387228861df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:06.145997 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd18753-37bd-4372-9102-3387228861df None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:06.146866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd18753-37bd-4372-9102-3387228861df None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:06.147650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd18753-37bd-4372-9102-3387228861df None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:06.170006 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd18753-37bd-4372-9102-3387228861df None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:44:06.180311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-efd18753-37bd-4372-9102-3387228861df None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 18 00:44:06.180311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:06.180311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:06.180311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:06.180311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:06.180311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:06.180311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:06.180311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:06.180311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:06.180311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:06.180311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:06.180311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:06.180311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:44:06.180311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:44:06.180311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:44:06.180311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:44:06.182129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:44:06.182129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:44:06.182129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:44:06.182129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:44:06.182129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:06.182129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:06.182129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:06.182129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:06.182129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:06.182129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:06.182129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:06.182129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:06.182129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:06.182129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:06.182129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:06.182129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:06.183854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:06.183854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:06.183854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:06.183854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:06.183854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:44:06.183854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:44:06.183854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:44:06.183854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:44:06.183854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:44:06.183854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:44:06.183854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 18 00:44:06.183854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:06.188570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 439/876] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Sep 18 00:44:05 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:06.194938 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2428aade-ccdf-4bef-abe5-12f32d878eb3 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:06.295379 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:06.313778 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2428aade-ccdf-4bef-abe5-12f32d878eb3 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:06.350429 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25b0009-6595-4941-9463-e12254b13e03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:06.351080 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25b0009-6595-4941-9463-e12254b13e03 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:06.351353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25b0009-6595-4941-9463-e12254b13e03 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:06.351610 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25b0009-6595-4941-9463-e12254b13e03 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:06.352281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25b0009-6595-4941-9463-e12254b13e03 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:06.352582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25b0009-6595-4941-9463-e12254b13e03 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:06.353995 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25b0009-6595-4941-9463-e12254b13e03 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:06.364928 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 438/877] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Sep 18 00:44:06 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:06.370937 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd18753-37bd-4372-9102-3387228861df None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:06.530331 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd18753-37bd-4372-9102-3387228861df None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:06.534158 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:06.592282 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8902951-3684-4bd1-a843-3ba701215cfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:06.593303 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8902951-3684-4bd1-a843-3ba701215cfc None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:06.594141 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8902951-3684-4bd1-a843-3ba701215cfc None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:06.594624 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8902951-3684-4bd1-a843-3ba701215cfc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:06.613911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8902951-3684-4bd1-a843-3ba701215cfc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:06.616545 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8902951-3684-4bd1-a843-3ba701215cfc None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:06.644746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-b8902951-3684-4bd1-a843-3ba701215cfc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:06.644746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:06.644746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:06.644746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:06.644746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:06.644746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:06.644746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:06.644746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:06.644746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:06.644746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:06.644746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:06.644746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:06.644746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:44:06.644746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:44:06.644746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:44:06.644746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:44:06.645753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:06.645753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:06.645753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:06.645753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:06.645753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:06.645753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:06.645753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:06.645753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:06.645753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:06.645753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:06.645753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:06.645753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:06.645753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:06.645753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:06.645753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:06.645753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:06.645753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:44:06.646687 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:44:06.646687 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:06.646687 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:06.647306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 440/878] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:06.653100 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25b0009-6595-4941-9463-e12254b13e03 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:06.764256 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25b0009-6595-4941-9463-e12254b13e03 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:06.801997 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ce82d9-a7e8-40ec-9798-83ac874d003a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:06.803015 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ce82d9-a7e8-40ec-9798-83ac874d003a None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:06.803531 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ce82d9-a7e8-40ec-9798-83ac874d003a None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:06.804003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ce82d9-a7e8-40ec-9798-83ac874d003a None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:06.804872 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ce82d9-a7e8-40ec-9798-83ac874d003a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:06.805378 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ce82d9-a7e8-40ec-9798-83ac874d003a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:06.806995 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ce82d9-a7e8-40ec-9798-83ac874d003a None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:06.815169 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:06.824109 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 439/879] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:44:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:06.829970 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8902951-3684-4bd1-a843-3ba701215cfc None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:06.980800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8902951-3684-4bd1-a843-3ba701215cfc None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:07.048244 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a98a13-f957-401a-896a-7c169def7dcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:07.049272 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a98a13-f957-401a-896a-7c169def7dcc None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:07.049750 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a98a13-f957-401a-896a-7c169def7dcc None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:07.050198 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a98a13-f957-401a-896a-7c169def7dcc None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:07.065412 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a98a13-f957-401a-896a-7c169def7dcc None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:07.068255 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a98a13-f957-401a-896a-7c169def7dcc None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:07.092957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-62a98a13-f957-401a-896a-7c169def7dcc None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 18 00:44:07.092957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:07.092957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:07.092957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:07.092957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:07.092957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:07.092957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:07.092957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:07.092957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:07.092957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:07.092957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:07.092957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:07.092957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:44:07.092957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:44:07.092957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:44:07.092957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:44:07.094532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:07.094532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:07.094532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:44:07.094532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:44:07.094532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:44:07.094532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:44:07.094532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:07.094532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:07.094532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:07.094532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:07.094532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:07.094532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:07.094532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:07.094532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:07.094532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:07.094532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:07.094532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:07.094532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:07.096053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:07.096053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:07.096053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:44:07.096053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:44:07.096053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:44:07.096053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:44:07.096053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:44:07.096053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:44:07.096053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 18 00:44:07.096053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:07.096053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 441/880] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Sep 18 00:44:06 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:07.104965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ce82d9-a7e8-40ec-9798-83ac874d003a None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:07.227642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ce82d9-a7e8-40ec-9798-83ac874d003a None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:07.267431 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68af36ce-f52a-49eb-801c-abb4bd546c25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:07.268461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68af36ce-f52a-49eb-801c-abb4bd546c25 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:07.268960 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68af36ce-f52a-49eb-801c-abb4bd546c25 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:07.269462 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68af36ce-f52a-49eb-801c-abb4bd546c25 None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:07.270474 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68af36ce-f52a-49eb-801c-abb4bd546c25 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:07.271029 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68af36ce-f52a-49eb-801c-abb4bd546c25 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:07.272737 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68af36ce-f52a-49eb-801c-abb4bd546c25 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:07.296350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 440/881] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Sep 18 00:44:07 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:07.303564 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a98a13-f957-401a-896a-7c169def7dcc None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:07.334318 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:07.425026 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a11f60-26ec-4af9-ad5d-ff297e7f5771 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:07.425487 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a11f60-26ec-4af9-ad5d-ff297e7f5771 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:07.425870 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a11f60-26ec-4af9-ad5d-ff297e7f5771 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:07.427961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 441/882] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:07.434094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5cfa4b-d2b7-4339-9ae8-f4a304e61486 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:07.434508 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5cfa4b-d2b7-4339-9ae8-f4a304e61486 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:07.434926 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5cfa4b-d2b7-4339-9ae8-f4a304e61486 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:07.456089 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-bb5cfa4b-d2b7-4339-9ae8-f4a304e61486 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:07.457136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-bb5cfa4b-d2b7-4339-9ae8-f4a304e61486 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:07.530668 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:07.532295 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a98a13-f957-401a-896a-7c169def7dcc None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:07.554802 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 442/883] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:07 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 18 00:44:07.601114 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d62581-d79b-45a7-b710-2c52d37b6179 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:07.603033 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d62581-d79b-45a7-b710-2c52d37b6179 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:07.603870 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d62581-d79b-45a7-b710-2c52d37b6179 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:07.604690 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d62581-d79b-45a7-b710-2c52d37b6179 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:07.722007 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e6a46a-b998-4358-84e6-1e9877c9e8f4 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:07.722439 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e6a46a-b998-4358-84e6-1e9877c9e8f4 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:07.722673 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e6a46a-b998-4358-84e6-1e9877c9e8f4 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:07.724236 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 443/884] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:07.730364 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e40590c-f789-4e1f-95b1-95a202bb9990 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:07.730781 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e40590c-f789-4e1f-95b1-95a202bb9990 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:07.731023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e40590c-f789-4e1f-95b1-95a202bb9990 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:07.740420 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d62581-d79b-45a7-b710-2c52d37b6179 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:07.743778 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d62581-d79b-45a7-b710-2c52d37b6179 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:07.749754 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-1e40590c-f789-4e1f-95b1-95a202bb9990 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:07.751012 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-1e40590c-f789-4e1f-95b1-95a202bb9990 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:07.773515 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:07.836149 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 444/885] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:07 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:07.847476 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e40590c-f789-4e1f-95b1-95a202bb9990 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:07.934678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 442/886] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:44:07 2022] GET /identity/v3/auth/tokens => generated 1262 bytes in 629 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 18 00:44:07.937126 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d62581-d79b-45a7-b710-2c52d37b6179 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:08.016501 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e40590c-f789-4e1f-95b1-95a202bb9990 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:08.041630 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:08.069259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a0e054-ed80-40e0-a79f-4a4074926811 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:08.071071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a0e054-ed80-40e0-a79f-4a4074926811 None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:08.071591 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a0e054-ed80-40e0-a79f-4a4074926811 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:08.072035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a0e054-ed80-40e0-a79f-4a4074926811 None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:08.072975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a0e054-ed80-40e0-a79f-4a4074926811 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:08.075454 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a0e054-ed80-40e0-a79f-4a4074926811 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:08.120720 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 445/887] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 18 00:44:07 2022] POST /identity/v3/services => generated 269 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:44:08.129119 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a0e054-ed80-40e0-a79f-4a4074926811 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:08.131337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d62581-d79b-45a7-b710-2c52d37b6179 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:08.172932 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:08.243597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bbc2d2-0065-4a6a-bf65-d564af9c0dcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:08.244850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bbc2d2-0065-4a6a-bf65-d564af9c0dcc None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:08.245353 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bbc2d2-0065-4a6a-bf65-d564af9c0dcc None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:08.245988 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bbc2d2-0065-4a6a-bf65-d564af9c0dcc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:08.268798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bbc2d2-0065-4a6a-bf65-d564af9c0dcc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:08.271898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bbc2d2-0065-4a6a-bf65-d564af9c0dcc None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:08.292843 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a0e054-ed80-40e0-a79f-4a4074926811 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:08.303488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-d2bbc2d2-0065-4a6a-bf65-d564af9c0dcc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:08.303488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:08.303488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:08.303488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:08.303488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:08.303488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:08.303488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:08.303488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:08.303488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:08.303488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:08.303488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:08.303488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:08.303488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:44:08.303488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:44:08.303488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:44:08.303488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:44:08.304479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:08.304479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:08.304479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:08.304479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:08.304479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:08.304479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:08.304479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:08.304479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:08.304479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:08.304479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:08.304479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:08.304479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:08.304479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:08.304479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:08.304479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:08.304479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:08.304479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:44:08.305387 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:44:08.305387 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:08.305387 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:08.306998 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 443/888] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 370 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:08.312057 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bbc2d2-0065-4a6a-bf65-d564af9c0dcc None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:08.333967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d11e7ab-fd22-4d8e-b773-51ea09c2a7a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:08.335052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d11e7ab-fd22-4d8e-b773-51ea09c2a7a0 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:08.335571 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d11e7ab-fd22-4d8e-b773-51ea09c2a7a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:08.336042 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d11e7ab-fd22-4d8e-b773-51ea09c2a7a0 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:08.336861 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d11e7ab-fd22-4d8e-b773-51ea09c2a7a0 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:08.338516 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d11e7ab-fd22-4d8e-b773-51ea09c2a7a0 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:08.348864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-9d11e7ab-fd22-4d8e-b773-51ea09c2a7a0 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 18 00:44:08.348864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:08.348864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:44:08.348864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:44:08.348864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:44:08.348864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:44:08.348864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:44:08.348864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:08.348864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 18 00:44:08.348864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:08.348864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:44:08.348864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:08.348864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:08.348864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:08.348864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:08.348864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:08.348864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:08.349901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:08.349901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:08.349901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:08.349901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:08.349901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:08.349901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:08.349901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:44:08.349901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:44:08.349901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:44:08.349901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:44:08.349901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:08.349901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:08.349901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:08.349901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:08.349901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:08.349901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:08.350757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:08.350757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:08.350757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:08.350757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:08.350757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:08.350757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:08.350757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:08.350757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:08.350757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:08.350757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:08.350757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:44:08.350757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:08.350757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 18 00:44:08.350757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:08.350757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 446/889] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 18 00:44:08 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:08.351682 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acba6eff-0259-4e79-a9e4-30e93106677e None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:08.351682 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acba6eff-0259-4e79-a9e4-30e93106677e None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:08.351826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acba6eff-0259-4e79-a9e4-30e93106677e None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:08.353088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 447/890] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:08 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:08.358601 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598d40b5-9b88-419e-ab6e-00e72a3c52ca None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:08.358862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598d40b5-9b88-419e-ab6e-00e72a3c52ca None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:08.359137 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598d40b5-9b88-419e-ab6e-00e72a3c52ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:08.380066 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-598d40b5-9b88-419e-ab6e-00e72a3c52ca None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:08.381274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-598d40b5-9b88-419e-ab6e-00e72a3c52ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:08.437119 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:08.440318 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bbc2d2-0065-4a6a-bf65-d564af9c0dcc None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:08.477855 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae23092-849b-47ea-8fbc-b836a4c6a7e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:08.478761 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae23092-849b-47ea-8fbc-b836a4c6a7e6 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:08.479890 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae23092-849b-47ea-8fbc-b836a4c6a7e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:08.480158 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae23092-849b-47ea-8fbc-b836a4c6a7e6 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:08.480815 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae23092-849b-47ea-8fbc-b836a4c6a7e6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:08.481120 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae23092-849b-47ea-8fbc-b836a4c6a7e6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:08.482663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae23092-849b-47ea-8fbc-b836a4c6a7e6 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:08.500304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 444/891] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:44:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:08.500304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae23092-849b-47ea-8fbc-b836a4c6a7e6 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:08.500581 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 448/892] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:08 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 18 00:44:08.502725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598d40b5-9b88-419e-ab6e-00e72a3c52ca None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:08.536891 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:08.669932 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598d40b5-9b88-419e-ab6e-00e72a3c52ca None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:08.676010 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae23092-849b-47ea-8fbc-b836a4c6a7e6 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:08.714905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93086269-e0b7-4852-9e56-de2e21242aea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:08.716023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93086269-e0b7-4852-9e56-de2e21242aea None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:08.716503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93086269-e0b7-4852-9e56-de2e21242aea None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:08.716974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93086269-e0b7-4852-9e56-de2e21242aea None admin] PATH_INFO: `/v3/projects/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:08.724657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93086269-e0b7-4852-9e56-de2e21242aea None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:08.726745 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93086269-e0b7-4852-9e56-de2e21242aea None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:08.739370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-93086269-e0b7-4852-9e56-de2e21242aea None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:44:08.739370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:08.739370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:08.739370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:08.739370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:08.739370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:08.739370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:08.739370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:08.739370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:08.739370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:08.739370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:08.739370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:08.739370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:44:08.739370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:44:08.739370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:44:08.740321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:44:08.740321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:08.740321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:08.740321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:08.740321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:08.740321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:08.740321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:08.740321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:08.740321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:08.740321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:08.740321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:08.740321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:08.740321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:08.740321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:08.740321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:08.740321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:08.740321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:08.741208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:44:08.741208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:44:08.741208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:44:08.741208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:44:08.741208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:44:08.741208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:44:08.741208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:44:08.741208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:08.741631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db358ad5-9a59-4df6-be39-e2adfe965aaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:08.741631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db358ad5-9a59-4df6-be39-e2adfe965aaa None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:08.741631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db358ad5-9a59-4df6-be39-e2adfe965aaa None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:08.741631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db358ad5-9a59-4df6-be39-e2adfe965aaa None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:08.742480 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db358ad5-9a59-4df6-be39-e2adfe965aaa None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:08.743024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db358ad5-9a59-4df6-be39-e2adfe965aaa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:08.746507 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db358ad5-9a59-4df6-be39-e2adfe965aaa None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:08.749508 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 449/893] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:44:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:08.750754 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93086269-e0b7-4852-9e56-de2e21242aea None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:08.762645 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 445/894] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 18 00:44:08 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:08.767596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db358ad5-9a59-4df6-be39-e2adfe965aaa None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:08.786451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:08.870569 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:08.870909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93086269-e0b7-4852-9e56-de2e21242aea None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:08.888807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db358ad5-9a59-4df6-be39-e2adfe965aaa None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:08.915757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2375df-5d94-46c3-aeb5-6ab99deca56e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:08.916954 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2375df-5d94-46c3-aeb5-6ab99deca56e None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:08.917182 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2375df-5d94-46c3-aeb5-6ab99deca56e None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:08.917543 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2375df-5d94-46c3-aeb5-6ab99deca56e None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:08.918898 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2375df-5d94-46c3-aeb5-6ab99deca56e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:08.919373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2375df-5d94-46c3-aeb5-6ab99deca56e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:08.920971 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2375df-5d94-46c3-aeb5-6ab99deca56e None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:08.933595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0e3ec0-c99e-44b1-8782-217e13ade344 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:08.934614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0e3ec0-c99e-44b1-8782-217e13ade344 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:08.934904 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0e3ec0-c99e-44b1-8782-217e13ade344 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:08.935149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0e3ec0-c99e-44b1-8782-217e13ade344 None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:08.935864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0e3ec0-c99e-44b1-8782-217e13ade344 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:08.936170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0e3ec0-c99e-44b1-8782-217e13ade344 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:08.937577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0e3ec0-c99e-44b1-8782-217e13ade344 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:08.941046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 450/895] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:44:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:08.942783 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2375df-5d94-46c3-aeb5-6ab99deca56e None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:08.948369 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 446/896] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 18 00:44:08 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:08.950169 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0e3ec0-c99e-44b1-8782-217e13ade344 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:09.091281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:09.104581 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2375df-5d94-46c3-aeb5-6ab99deca56e None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:09.121077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0e3ec0-c99e-44b1-8782-217e13ade344 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:09.145489 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f989d9c-b414-4943-b7f6-c1c1680c35bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:09.146319 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f989d9c-b414-4943-b7f6-c1c1680c35bb None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:09.146752 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f989d9c-b414-4943-b7f6-c1c1680c35bb None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:09.147209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f989d9c-b414-4943-b7f6-c1c1680c35bb None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:09.148111 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f989d9c-b414-4943-b7f6-c1c1680c35bb None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:09.149708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f989d9c-b414-4943-b7f6-c1c1680c35bb None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:09.161138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-5f989d9c-b414-4943-b7f6-c1c1680c35bb None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 18 00:44:09.161138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:09.161138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:44:09.161138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:44:09.161138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:44:09.161138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:44:09.161138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:44:09.161138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:09.161138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 18 00:44:09.161138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:09.161138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:44:09.161138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:09.161138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:09.161138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:09.161138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:09.161138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:09.161138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:09.162202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:09.162202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:09.162202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:09.162202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:09.162202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:09.162202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:09.162202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:44:09.162202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:44:09.162202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:44:09.162202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:44:09.162202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:09.162202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:09.162202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:09.162202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:09.162202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:09.162202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:09.163067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:09.163067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:09.163067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:09.163067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:09.163067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:09.163067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:09.163067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:09.163067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:09.163067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:09.163067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:09.163067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:44:09.163067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:09.163067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 18 00:44:09.163067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:09.163067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 451/897] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Sep 18 00:44:08 2022] GET /identity/v3/services/object-store => generated 93 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:09.163849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f989d9c-b414-4943-b7f6-c1c1680c35bb None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:09.175507 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051f7873-bcb3-4ec7-baa3-c5dddc2f6d11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:09.176492 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051f7873-bcb3-4ec7-baa3-c5dddc2f6d11 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:09.176835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051f7873-bcb3-4ec7-baa3-c5dddc2f6d11 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:09.177176 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051f7873-bcb3-4ec7-baa3-c5dddc2f6d11 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:09.178009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051f7873-bcb3-4ec7-baa3-c5dddc2f6d11 None admin] RBAC: Adding query filter params (role.id=88c42ae856ff4e2d8a42a5b1d50b97f9, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=3e31dce69c344c9aa4a55f31b9c735df) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:09.178386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051f7873-bcb3-4ec7-baa3-c5dddc2f6d11 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:09.180548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051f7873-bcb3-4ec7-baa3-c5dddc2f6d11 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:09.189324 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 447/898] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:44:08 2022] GET /identity/v3/role_assignments?user.id=3e31dce69c344c9aa4a55f31b9c735df&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=88c42ae856ff4e2d8a42a5b1d50b97f9 => generated 259 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:09.190702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051f7873-bcb3-4ec7-baa3-c5dddc2f6d11 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:09.225357 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:09.378640 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f989d9c-b414-4943-b7f6-c1c1680c35bb None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:09.389607 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051f7873-bcb3-4ec7-baa3-c5dddc2f6d11 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:09.446049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66183f86-0376-4fcd-bc9d-2106a1dceca2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:09.447114 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66183f86-0376-4fcd-bc9d-2106a1dceca2 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:09.447401 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66183f86-0376-4fcd-bc9d-2106a1dceca2 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:09.447944 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66183f86-0376-4fcd-bc9d-2106a1dceca2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:09.448834 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66183f86-0376-4fcd-bc9d-2106a1dceca2 None admin] RBAC: Adding query filter params (interface=public, service_id=563d18861970452c8d0d86811a22a119, region_id=RegionOne) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:09.449075 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66183f86-0376-4fcd-bc9d-2106a1dceca2 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:09.450723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66183f86-0376-4fcd-bc9d-2106a1dceca2 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:09.455278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c38066d-c2ea-4bbe-947e-ece000cfa831 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:09.456705 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c38066d-c2ea-4bbe-947e-ece000cfa831 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:09.457265 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c38066d-c2ea-4bbe-947e-ece000cfa831 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:09.457747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c38066d-c2ea-4bbe-947e-ece000cfa831 None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:09.458912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c38066d-c2ea-4bbe-947e-ece000cfa831 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:09.459440 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c38066d-c2ea-4bbe-947e-ece000cfa831 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:09.461951 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c38066d-c2ea-4bbe-947e-ece000cfa831 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:09.462843 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 452/899] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Sep 18 00:44:09 2022] GET /identity/v3/endpoints?service_id=563d18861970452c8d0d86811a22a119&interface=public®ion_id=RegionOne => generated 194 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:09.464621 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66183f86-0376-4fcd-bc9d-2106a1dceca2 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:09.473274 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 448/900] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Sep 18 00:44:09 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:09.482145 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c38066d-c2ea-4bbe-947e-ece000cfa831 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:09.492091 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:09.593863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66183f86-0376-4fcd-bc9d-2106a1dceca2 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:09.638976 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f58687-f593-473f-9839-c6e85b0a7ca5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:09.639499 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f58687-f593-473f-9839-c6e85b0a7ca5 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:09.639780 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f58687-f593-473f-9839-c6e85b0a7ca5 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:09.640062 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f58687-f593-473f-9839-c6e85b0a7ca5 None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:09.640681 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f58687-f593-473f-9839-c6e85b0a7ca5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:09.642086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f58687-f593-473f-9839-c6e85b0a7ca5 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:09.654763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 453/901] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Sep 18 00:44:09 2022] GET /identity/v3/services => generated 1268 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:44:09.669372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:09.699329 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c38066d-c2ea-4bbe-947e-ece000cfa831 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:09.768003 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461cf09b-3749-45b2-9f4c-23de84ac7996 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:09.769082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461cf09b-3749-45b2-9f4c-23de84ac7996 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:09.769564 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461cf09b-3749-45b2-9f4c-23de84ac7996 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:09.770035 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461cf09b-3749-45b2-9f4c-23de84ac7996 None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:09.771303 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461cf09b-3749-45b2-9f4c-23de84ac7996 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:09.771823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461cf09b-3749-45b2-9f4c-23de84ac7996 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:09.774436 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461cf09b-3749-45b2-9f4c-23de84ac7996 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:09.783885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 449/902] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Sep 18 00:44:09 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:11.140243 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7259dc78-b016-498c-a6f0-8503aefb41e4 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:11.141115 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7259dc78-b016-498c-a6f0-8503aefb41e4 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:11.141566 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7259dc78-b016-498c-a6f0-8503aefb41e4 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:11.143421 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 454/903] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:11.149587 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa998554-ae62-46bb-8775-43fa5c54942d None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:11.150803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa998554-ae62-46bb-8775-43fa5c54942d None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:11.151386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa998554-ae62-46bb-8775-43fa5c54942d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:11.171335 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-fa998554-ae62-46bb-8775-43fa5c54942d None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:11.172903 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-fa998554-ae62-46bb-8775-43fa5c54942d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:11.306613 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 450/904] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:11 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:11.687535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5399241-5de9-4d71-897c-01c1375528c6 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:11.687947 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5399241-5de9-4d71-897c-01c1375528c6 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:11.688224 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5399241-5de9-4d71-897c-01c1375528c6 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:11.690046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 455/905] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:11.695858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d2f594-723e-4cb6-befe-6423be00e5e4 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:11.696164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d2f594-723e-4cb6-befe-6423be00e5e4 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:11.696406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d2f594-723e-4cb6-befe-6423be00e5e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:11.716519 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5399241-5de9-4d71-897c-01c1375528c6 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:11.717268 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-97d2f594-723e-4cb6-befe-6423be00e5e4 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:11.718441 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-97d2f594-723e-4cb6-befe-6423be00e5e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:11.810987 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:11.818358 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 451/906] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:11 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:11.837968 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5399241-5de9-4d71-897c-01c1375528c6 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:11.875917 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2072616c-a352-4f67-b1c7-99ba634d3541 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:11.876564 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2072616c-a352-4f67-b1c7-99ba634d3541 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:11.876843 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2072616c-a352-4f67-b1c7-99ba634d3541 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:11.877170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2072616c-a352-4f67-b1c7-99ba634d3541 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:11.877763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2072616c-a352-4f67-b1c7-99ba634d3541 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:11.879250 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2072616c-a352-4f67-b1c7-99ba634d3541 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:11.888959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-2072616c-a352-4f67-b1c7-99ba634d3541 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 18 00:44:11.888959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:11.888959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:44:11.888959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:44:11.888959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:44:11.888959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:44:11.888959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:44:11.888959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:11.888959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 18 00:44:11.888959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:11.888959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:44:11.888959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:11.888959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:11.888959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:11.888959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:11.888959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:11.889965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:11.889965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:11.889965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:11.889965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:11.889965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:11.889965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:11.889965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:11.889965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:44:11.889965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:44:11.889965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:44:11.889965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:44:11.889965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:11.889965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:11.889965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:11.889965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:11.889965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:11.889965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:11.890885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:11.890885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:11.890885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:11.890885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:11.890885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:11.890885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:11.890885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:11.890885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:11.890885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:11.890885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:11.890885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:44:11.890885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:11.890885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 18 00:44:11.890885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:11.894655 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 456/907] 10.222.0.42 () {56 vars in 1132 bytes} [Sun Sep 18 00:44:11 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:11.896922 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d2f594-723e-4cb6-befe-6423be00e5e4 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:11.899896 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:12.022666 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d2f594-723e-4cb6-befe-6423be00e5e4 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:12.061499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc5ecb3-6bab-492c-9637-2dbc67e8b5eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:12.062559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc5ecb3-6bab-492c-9637-2dbc67e8b5eb None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:12.063042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc5ecb3-6bab-492c-9637-2dbc67e8b5eb None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:12.063514 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc5ecb3-6bab-492c-9637-2dbc67e8b5eb None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:12.064377 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc5ecb3-6bab-492c-9637-2dbc67e8b5eb None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:12.064883 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc5ecb3-6bab-492c-9637-2dbc67e8b5eb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:12.066505 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc5ecb3-6bab-492c-9637-2dbc67e8b5eb None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:12.072424 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 452/908] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Sep 18 00:44:11 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:12.081286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2072616c-a352-4f67-b1c7-99ba634d3541 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:12.115735 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc5ecb3-6bab-492c-9637-2dbc67e8b5eb None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:12.190800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:12.199263 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2072616c-a352-4f67-b1c7-99ba634d3541 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:12.236312 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc5ecb3-6bab-492c-9637-2dbc67e8b5eb None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:12.257020 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618f0356-e879-4d74-b185-1753abfa172c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:12.258270 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618f0356-e879-4d74-b185-1753abfa172c None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:12.258845 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618f0356-e879-4d74-b185-1753abfa172c None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:12.259784 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618f0356-e879-4d74-b185-1753abfa172c None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:12.260701 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618f0356-e879-4d74-b185-1753abfa172c None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:12.261173 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618f0356-e879-4d74-b185-1753abfa172c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:12.262819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618f0356-e879-4d74-b185-1753abfa172c None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:12.277291 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 457/909] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Sep 18 00:44:12 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:12.280402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618f0356-e879-4d74-b185-1753abfa172c None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:12.284387 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f3d893-8133-43fa-b1a2-b18659d73481 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:12.285188 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f3d893-8133-43fa-b1a2-b18659d73481 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:12.285573 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f3d893-8133-43fa-b1a2-b18659d73481 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:12.285941 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f3d893-8133-43fa-b1a2-b18659d73481 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:12.299052 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f3d893-8133-43fa-b1a2-b18659d73481 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:44:12.299453 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:12.305585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-c7f3d893-8133-43fa-b1a2-b18659d73481 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 18 00:44:12.305585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:12.305585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:12.305585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:12.305585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:12.305585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:12.305585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:12.305585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:12.305585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:12.305585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:12.305585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:12.305585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:12.305585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:44:12.305585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:44:12.305585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:44:12.305585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:44:12.306888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:44:12.306888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:44:12.306888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:44:12.306888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:44:12.306888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:12.306888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:12.306888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:12.306888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:12.306888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:12.306888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:12.306888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:12.306888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:12.306888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:12.306888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:12.306888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:12.306888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:12.307982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:12.307982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:12.307982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:12.307982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:12.307982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:44:12.307982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:44:12.307982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:44:12.307982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:44:12.307982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:44:12.307982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:44:12.307982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 18 00:44:12.307982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:12.312535 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 453/910] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Sep 18 00:44:12 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:12.313554 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f3d893-8133-43fa-b1a2-b18659d73481 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:12.424376 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618f0356-e879-4d74-b185-1753abfa172c None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:12.455559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f3d893-8133-43fa-b1a2-b18659d73481 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:12.467582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec359ab-5016-4b5b-8685-8c9c5fcbd752 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:12.468537 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec359ab-5016-4b5b-8685-8c9c5fcbd752 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:12.469084 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec359ab-5016-4b5b-8685-8c9c5fcbd752 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:12.469540 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec359ab-5016-4b5b-8685-8c9c5fcbd752 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:12.470571 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec359ab-5016-4b5b-8685-8c9c5fcbd752 None admin] RBAC: Adding query filter params (interface=public, service_id=76cb633d4a194f5baaa30eaa0175b251, region_id=RegionOne) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:12.471160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec359ab-5016-4b5b-8685-8c9c5fcbd752 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:12.473063 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec359ab-5016-4b5b-8685-8c9c5fcbd752 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:12.485006 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 458/911] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Sep 18 00:44:12 2022] GET /identity/v3/endpoints?service_id=76cb633d4a194f5baaa30eaa0175b251&interface=public®ion_id=RegionOne => generated 194 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:12.486797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec359ab-5016-4b5b-8685-8c9c5fcbd752 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:12.494181 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:12.507373 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8770dd-af03-4635-9c15-67b8d58f0967 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:12.508684 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8770dd-af03-4635-9c15-67b8d58f0967 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:12.508990 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8770dd-af03-4635-9c15-67b8d58f0967 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:12.509267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8770dd-af03-4635-9c15-67b8d58f0967 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:12.509990 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8770dd-af03-4635-9c15-67b8d58f0967 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:12.510370 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8770dd-af03-4635-9c15-67b8d58f0967 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:12.512776 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8770dd-af03-4635-9c15-67b8d58f0967 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:12.529469 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 454/912] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Sep 18 00:44:12 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:12.530594 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8770dd-af03-4635-9c15-67b8d58f0967 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:12.605106 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:12.638078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec359ab-5016-4b5b-8685-8c9c5fcbd752 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:12.681938 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8770dd-af03-4635-9c15-67b8d58f0967 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:12.682807 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17876dc4-627c-4f1e-b2d5-81f67cd91a8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:12.683218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17876dc4-627c-4f1e-b2d5-81f67cd91a8a None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:12.683501 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17876dc4-627c-4f1e-b2d5-81f67cd91a8a None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:12.683775 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17876dc4-627c-4f1e-b2d5-81f67cd91a8a None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:12.684393 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17876dc4-627c-4f1e-b2d5-81f67cd91a8a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:12.686135 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17876dc4-627c-4f1e-b2d5-81f67cd91a8a None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:12.702505 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 459/913] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Sep 18 00:44:12 2022] GET /identity/v3/services => generated 1268 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:44:12.728829 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f994d7-a5ba-4d1b-9cdd-abaa5260ed79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:12.730023 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f994d7-a5ba-4d1b-9cdd-abaa5260ed79 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:12.730303 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f994d7-a5ba-4d1b-9cdd-abaa5260ed79 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:12.730649 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f994d7-a5ba-4d1b-9cdd-abaa5260ed79 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:12.743760 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f994d7-a5ba-4d1b-9cdd-abaa5260ed79 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:12.745436 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f994d7-a5ba-4d1b-9cdd-abaa5260ed79 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:12.763579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-37f994d7-a5ba-4d1b-9cdd-abaa5260ed79 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:12.763579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:12.763579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:12.763579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:12.763579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:12.763579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:12.763579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:12.763579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:12.763579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:12.763579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:12.763579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:12.763579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:12.763579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:44:12.763579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:44:12.763579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:44:12.763579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:44:12.764593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:12.764593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:12.764593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:12.764593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:12.764593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:12.764593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:12.764593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:12.764593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:12.764593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:12.764593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:12.764593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:12.764593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:12.764593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:12.764593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:12.764593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:12.764593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:12.764593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:44:12.765946 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:44:12.765946 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:12.765946 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:12.765946 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 455/914] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:12.771247 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17876dc4-627c-4f1e-b2d5-81f67cd91a8a None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:12.901705 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17876dc4-627c-4f1e-b2d5-81f67cd91a8a None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:12.941922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb2d294-7753-4607-8bda-e1fc31e4a0fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:12.942597 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb2d294-7753-4607-8bda-e1fc31e4a0fc None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:12.942879 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb2d294-7753-4607-8bda-e1fc31e4a0fc None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:12.943160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb2d294-7753-4607-8bda-e1fc31e4a0fc None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:12.943860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb2d294-7753-4607-8bda-e1fc31e4a0fc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:12.944167 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb2d294-7753-4607-8bda-e1fc31e4a0fc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:12.945682 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb2d294-7753-4607-8bda-e1fc31e4a0fc None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:12.955511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:12.962293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 460/915] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:44:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:12.969158 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f994d7-a5ba-4d1b-9cdd-abaa5260ed79 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:13.023206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:13.096122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f994d7-a5ba-4d1b-9cdd-abaa5260ed79 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:13.138716 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd791a1b-7a56-463a-a214-0ab1bdc16a7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:13.139676 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd791a1b-7a56-463a-a214-0ab1bdc16a7b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:13.140158 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd791a1b-7a56-463a-a214-0ab1bdc16a7b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:13.140611 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd791a1b-7a56-463a-a214-0ab1bdc16a7b None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:13.149649 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd791a1b-7a56-463a-a214-0ab1bdc16a7b None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:13.152078 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd791a1b-7a56-463a-a214-0ab1bdc16a7b None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:13.165068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-fd791a1b-7a56-463a-a214-0ab1bdc16a7b None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 18 00:44:13.165068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:13.165068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:13.165068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:13.165068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:13.165068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:13.165068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:13.165068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:13.165068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:13.165068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:13.165068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:13.165068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:13.165068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:44:13.165068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:44:13.165068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:44:13.165068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:44:13.166065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:13.166065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:13.166065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:44:13.166065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:44:13.166065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:44:13.166065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:44:13.166065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:13.166065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:13.166065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:13.166065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:13.166065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:13.166065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:13.166065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:13.166065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:13.166065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:13.166065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:13.166065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:13.166065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:13.167133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:13.167133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:13.167133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:44:13.167133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:44:13.167133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:44:13.167133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:44:13.167133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:44:13.167133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:44:13.167133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 18 00:44:13.167133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:13.169398 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 456/916] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Sep 18 00:44:12 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:13.175448 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb2d294-7753-4607-8bda-e1fc31e4a0fc None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:13.283362 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe222fc-16ce-47ce-99ff-f630ccad82de None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:13.284407 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe222fc-16ce-47ce-99ff-f630ccad82de None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:13.284898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe222fc-16ce-47ce-99ff-f630ccad82de None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:13.287436 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 457/917] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:13 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:13.295751 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78403c49-81da-4a64-8476-ba72d99d3647 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:13.296226 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78403c49-81da-4a64-8476-ba72d99d3647 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:13.296938 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78403c49-81da-4a64-8476-ba72d99d3647 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:13.299864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb2d294-7753-4607-8bda-e1fc31e4a0fc None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:13.327732 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-78403c49-81da-4a64-8476-ba72d99d3647 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:13.329720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-78403c49-81da-4a64-8476-ba72d99d3647 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:13.343276 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0574f920-3dda-4a66-ba2b-2028213fb827 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:13.344303 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0574f920-3dda-4a66-ba2b-2028213fb827 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:13.345383 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0574f920-3dda-4a66-ba2b-2028213fb827 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:13.345813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0574f920-3dda-4a66-ba2b-2028213fb827 None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:13.346796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0574f920-3dda-4a66-ba2b-2028213fb827 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:13.347278 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0574f920-3dda-4a66-ba2b-2028213fb827 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:13.349505 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0574f920-3dda-4a66-ba2b-2028213fb827 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:13.379199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 461/918] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Sep 18 00:44:13 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:13.382559 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0574f920-3dda-4a66-ba2b-2028213fb827 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:13.441638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 458/919] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:13 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:13.483547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:13.543986 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0574f920-3dda-4a66-ba2b-2028213fb827 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:13.577048 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c741ba3-2d3d-406e-a257-a1ca0555aa97 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:13.577526 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c741ba3-2d3d-406e-a257-a1ca0555aa97 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:13.577992 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c741ba3-2d3d-406e-a257-a1ca0555aa97 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:13.583449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bc4c21-9860-40fd-bd5f-c8478894d542 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:13.584385 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bc4c21-9860-40fd-bd5f-c8478894d542 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:13.584846 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bc4c21-9860-40fd-bd5f-c8478894d542 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:13.585295 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bc4c21-9860-40fd-bd5f-c8478894d542 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:13.586844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 459/920] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:13 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:13.589176 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6454b34-8b94-4a72-ab06-3a91ebcb8957 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:13.589650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6454b34-8b94-4a72-ab06-3a91ebcb8957 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:13.590133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6454b34-8b94-4a72-ab06-3a91ebcb8957 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:13.598011 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4bc4c21-9860-40fd-bd5f-c8478894d542 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:13.599769 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4bc4c21-9860-40fd-bd5f-c8478894d542 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:13.619071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-a4bc4c21-9860-40fd-bd5f-c8478894d542 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:13.619071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:13.619071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:13.619071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:13.619071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:13.619071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:13.619071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:13.619071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:13.619071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:13.619071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:13.619071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:13.619071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:13.619071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:44:13.619071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:44:13.619071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:44:13.619071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:44:13.620349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:13.620349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:13.620349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:13.620349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:13.620349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:13.620349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:13.620349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:13.620349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:13.620349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:13.620349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:13.620349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:13.620349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:13.620349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:13.620349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:13.620349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:13.620349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:13.620349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:44:13.621436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:44:13.621436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:13.621436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:13.623221 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 462/921] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:13.627856 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-d6454b34-8b94-4a72-ab06-3a91ebcb8957 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:13.627856 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d6454b34-8b94-4a72-ab06-3a91ebcb8957 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:13.629245 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bc4c21-9860-40fd-bd5f-c8478894d542 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:13.672978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:13.754057 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 460/922] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:13 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 18 00:44:13.755273 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6454b34-8b94-4a72-ab06-3a91ebcb8957 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:13.765195 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bc4c21-9860-40fd-bd5f-c8478894d542 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:13.782472 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:13.828515 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f82fd8-dad7-45b5-9a45-3b1d602aa532 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:13.829586 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f82fd8-dad7-45b5-9a45-3b1d602aa532 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:13.829962 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f82fd8-dad7-45b5-9a45-3b1d602aa532 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:13.830427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f82fd8-dad7-45b5-9a45-3b1d602aa532 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:13.831759 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f82fd8-dad7-45b5-9a45-3b1d602aa532 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:13.832306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f82fd8-dad7-45b5-9a45-3b1d602aa532 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:13.834957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f82fd8-dad7-45b5-9a45-3b1d602aa532 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:13.860327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 463/923] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:44:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:13.862101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f82fd8-dad7-45b5-9a45-3b1d602aa532 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:13.875245 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6454b34-8b94-4a72-ab06-3a91ebcb8957 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:13.913573 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb59f4c0-61a2-45ac-bed5-a43121a75494 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:13.914616 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb59f4c0-61a2-45ac-bed5-a43121a75494 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:13.915102 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb59f4c0-61a2-45ac-bed5-a43121a75494 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:13.915978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb59f4c0-61a2-45ac-bed5-a43121a75494 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:13.917350 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb59f4c0-61a2-45ac-bed5-a43121a75494 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:13.919296 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb59f4c0-61a2-45ac-bed5-a43121a75494 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:13.930823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-fb59f4c0-61a2-45ac-bed5-a43121a75494 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 18 00:44:13.930823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:13.930823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:44:13.930823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:44:13.930823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:44:13.930823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:44:13.930823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:44:13.930823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:13.930823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 18 00:44:13.930823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:13.930823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:44:13.930823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:13.930823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:13.930823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:13.930823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:13.930823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:13.930823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:13.931901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:13.931901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:13.931901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:13.931901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:13.931901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:13.931901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:13.931901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:44:13.931901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:44:13.931901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:44:13.931901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:44:13.931901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:13.931901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:13.931901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:13.931901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:13.931901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:13.931901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:13.933293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:13.933293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:13.933293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:13.933293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:13.933293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:13.933293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:13.933293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:13.933293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:13.933293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:13.933293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:13.933293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:44:13.933293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:13.933293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 18 00:44:13.933293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:13.937166 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 461/924] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 18 00:44:13 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:13.943410 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb59f4c0-61a2-45ac-bed5-a43121a75494 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:13.980658 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:14.050842 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f82fd8-dad7-45b5-9a45-3b1d602aa532 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:14.086155 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb59f4c0-61a2-45ac-bed5-a43121a75494 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:14.097481 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1acd0c-0f0d-4f65-ab84-c439db460ddd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:14.098915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1acd0c-0f0d-4f65-ab84-c439db460ddd None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:14.099322 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1acd0c-0f0d-4f65-ab84-c439db460ddd None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:14.099714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1acd0c-0f0d-4f65-ab84-c439db460ddd None admin] PATH_INFO: `/v3/projects/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:14.109257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a1acd0c-0f0d-4f65-ab84-c439db460ddd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:14.110862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a1acd0c-0f0d-4f65-ab84-c439db460ddd None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:14.123914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-1a1acd0c-0f0d-4f65-ab84-c439db460ddd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:44:14.123914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:14.123914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:14.123914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:14.123914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:14.123914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:14.123914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:14.123914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:14.123914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:14.123914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:14.123914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:14.123914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:14.123914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:44:14.123914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:44:14.123914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:44:14.125557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:44:14.125557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:14.125557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:14.125557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:14.125557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:14.125557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:14.125557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:14.125557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:14.125557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:14.125557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:14.125557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:14.125557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:14.125557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:14.125557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:14.125557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:14.125557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:14.125557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:14.127281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:44:14.127281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:44:14.127281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:44:14.127281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:44:14.127281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:44:14.127281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:44:14.127281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:44:14.127281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:14.127281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 464/925] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:44:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:14.127281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1acd0c-0f0d-4f65-ab84-c439db460ddd None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:14.129138 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78548055-da52-41df-a225-76112abd014f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:14.130277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78548055-da52-41df-a225-76112abd014f None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:14.130597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78548055-da52-41df-a225-76112abd014f None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:14.130838 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78548055-da52-41df-a225-76112abd014f None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:14.131521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78548055-da52-41df-a225-76112abd014f None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:14.131822 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78548055-da52-41df-a225-76112abd014f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:14.133213 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78548055-da52-41df-a225-76112abd014f None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:14.139916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 462/926] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 18 00:44:13 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:14.144740 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78548055-da52-41df-a225-76112abd014f None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:14.176155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:14.252571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:14.268814 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78548055-da52-41df-a225-76112abd014f None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:14.318933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b289de5-8454-4110-97ef-37c36819b55d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:14.320570 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1acd0c-0f0d-4f65-ab84-c439db460ddd None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:14.322010 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b289de5-8454-4110-97ef-37c36819b55d None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:14.323350 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b289de5-8454-4110-97ef-37c36819b55d None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:14.324037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b289de5-8454-4110-97ef-37c36819b55d None admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:14.325577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b289de5-8454-4110-97ef-37c36819b55d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:14.326130 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b289de5-8454-4110-97ef-37c36819b55d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:14.328913 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b289de5-8454-4110-97ef-37c36819b55d None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:14.356650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 463/927] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:44:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:14.356650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b289de5-8454-4110-97ef-37c36819b55d None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:14.381319 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff4bf64-a36f-44aa-a966-243a82ea88d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:14.382410 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff4bf64-a36f-44aa-a966-243a82ea88d2 None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:14.382850 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff4bf64-a36f-44aa-a966-243a82ea88d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:14.383326 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff4bf64-a36f-44aa-a966-243a82ea88d2 None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:14.384475 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff4bf64-a36f-44aa-a966-243a82ea88d2 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:14.387509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff4bf64-a36f-44aa-a966-243a82ea88d2 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:14.414525 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 465/928] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 18 00:44:14 2022] POST /identity/v3/services => generated 241 bytes in 287 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:44:14.414525 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff4bf64-a36f-44aa-a966-243a82ea88d2 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:14.555747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b289de5-8454-4110-97ef-37c36819b55d None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:14.603886 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485b2c5f-9519-4e4b-939d-f0a8fdbdb735 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:14.604474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485b2c5f-9519-4e4b-939d-f0a8fdbdb735 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:14.604752 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485b2c5f-9519-4e4b-939d-f0a8fdbdb735 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:14.605022 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485b2c5f-9519-4e4b-939d-f0a8fdbdb735 None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:14.605711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485b2c5f-9519-4e4b-939d-f0a8fdbdb735 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:14.606013 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485b2c5f-9519-4e4b-939d-f0a8fdbdb735 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:14.607519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485b2c5f-9519-4e4b-939d-f0a8fdbdb735 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:14.608448 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:14.620259 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 464/929] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 18 00:44:14 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:14.622654 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485b2c5f-9519-4e4b-939d-f0a8fdbdb735 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:14.632267 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff4bf64-a36f-44aa-a966-243a82ea88d2 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:14.650418 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:14.683510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8735cb4c-3b44-4d27-b74a-be74a16bdc9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:14.684538 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8735cb4c-3b44-4d27-b74a-be74a16bdc9c None admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:14.684814 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8735cb4c-3b44-4d27-b74a-be74a16bdc9c None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:14.685090 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8735cb4c-3b44-4d27-b74a-be74a16bdc9c None admin] PATH_INFO: `/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/3e31dce69c344c9aa4a55f31b9c735df/roles/88c42ae856ff4e2d8a42a5b1d50b97f9` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:14.688383 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8735cb4c-3b44-4d27-b74a-be74a16bdc9c None admin] RBAC: Authorizing `identity:create_grant(project_id=c520a8da3dc54adba4f7cc04ef42e367, user_id=3e31dce69c344c9aa4a55f31b9c735df, role_id=88c42ae856ff4e2d8a42a5b1d50b97f9)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:14.689869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8735cb4c-3b44-4d27-b74a-be74a16bdc9c None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:14.717806 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 466/930] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:44:14 2022] PUT /identity/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/3e31dce69c344c9aa4a55f31b9c735df/roles/88c42ae856ff4e2d8a42a5b1d50b97f9 => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:44:14.844617 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485b2c5f-9519-4e4b-939d-f0a8fdbdb735 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:14.913084 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1105a12-55ac-4967-85a8-442f41f6b943 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:14.914611 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1105a12-55ac-4967-85a8-442f41f6b943 None admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:14.915384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1105a12-55ac-4967-85a8-442f41f6b943 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:14.916157 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1105a12-55ac-4967-85a8-442f41f6b943 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:14.917736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1105a12-55ac-4967-85a8-442f41f6b943 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:14.920665 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1105a12-55ac-4967-85a8-442f41f6b943 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:14.959400 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 465/931] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:44:14 2022] POST /identity/v3/endpoints => generated 346 bytes in 338 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:44:15.377297 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94b0352-6467-4370-b40c-b130bf00a9a0 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:15.378080 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94b0352-6467-4370-b40c-b130bf00a9a0 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:15.378215 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94b0352-6467-4370-b40c-b130bf00a9a0 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:15.381207 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 467/932] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:15 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:15.390915 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9bc86a-8e28-4c43-8f49-e58ecdf66cc9 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:15.391718 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9bc86a-8e28-4c43-8f49-e58ecdf66cc9 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:15.391863 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9bc86a-8e28-4c43-8f49-e58ecdf66cc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:15.423100 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-5b9bc86a-8e28-4c43-8f49-e58ecdf66cc9 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:15.425336 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-5b9bc86a-8e28-4c43-8f49-e58ecdf66cc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:15.542867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:15.561955 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 466/933] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:15 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:16.054168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94b0352-6467-4370-b40c-b130bf00a9a0 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:16.252122 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94b0352-6467-4370-b40c-b130bf00a9a0 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:16.316573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c646df-aa0c-43ca-bd01-e041263204f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:16.317540 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c646df-aa0c-43ca-bd01-e041263204f9 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:16.318010 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c646df-aa0c-43ca-bd01-e041263204f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:16.318487 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c646df-aa0c-43ca-bd01-e041263204f9 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:16.319536 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c646df-aa0c-43ca-bd01-e041263204f9 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:16.322109 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c646df-aa0c-43ca-bd01-e041263204f9 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:16.340174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-59c646df-aa0c-43ca-bd01-e041263204f9 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 18 00:44:16.340174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:16.340174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:44:16.340174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:44:16.340174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:44:16.340174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:44:16.340174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:44:16.340174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:16.340174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 18 00:44:16.340174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:16.340174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:44:16.340174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:16.340174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:16.340174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:16.340174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:16.340174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:16.341184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:16.341184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:16.341184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:16.341184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:16.341184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:16.341184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:16.341184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:16.341184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:44:16.341184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:44:16.341184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:44:16.341184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:44:16.341184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:16.341184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:16.341184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:16.341184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:16.341184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:16.341184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:16.342047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:16.342047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:16.342047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:16.342047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:16.342047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:16.342047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:16.342047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:16.342047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:16.342047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:16.342047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:16.342047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:44:16.342047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:16.342047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 18 00:44:16.342047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:16.342785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 468/934] 10.222.0.42 () {56 vars in 1132 bytes} [Sun Sep 18 00:44:16 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:16.349241 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9bc86a-8e28-4c43-8f49-e58ecdf66cc9 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:16.535697 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9bc86a-8e28-4c43-8f49-e58ecdf66cc9 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:16.604767 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d065ee6-c74d-454d-87e8-0e8f152cc692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:16.605721 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d065ee6-c74d-454d-87e8-0e8f152cc692 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:16.606195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d065ee6-c74d-454d-87e8-0e8f152cc692 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:16.606694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d065ee6-c74d-454d-87e8-0e8f152cc692 None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:16.607828 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d065ee6-c74d-454d-87e8-0e8f152cc692 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:16.608346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d065ee6-c74d-454d-87e8-0e8f152cc692 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:16.610877 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d065ee6-c74d-454d-87e8-0e8f152cc692 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:16.620597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 467/935] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Sep 18 00:44:16 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:16.627835 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c646df-aa0c-43ca-bd01-e041263204f9 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:16.656171 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:16.825081 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c646df-aa0c-43ca-bd01-e041263204f9 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:16.890523 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3453c279-0193-43ee-bab8-6ec61751d455 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:16.891924 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3453c279-0193-43ee-bab8-6ec61751d455 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:16.892808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3453c279-0193-43ee-bab8-6ec61751d455 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:16.893520 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3453c279-0193-43ee-bab8-6ec61751d455 None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:16.894983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3453c279-0193-43ee-bab8-6ec61751d455 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:16.895713 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3453c279-0193-43ee-bab8-6ec61751d455 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:16.898448 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3453c279-0193-43ee-bab8-6ec61751d455 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:16.909179 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 469/936] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Sep 18 00:44:16 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:16.918829 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d065ee6-c74d-454d-87e8-0e8f152cc692 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:17.024855 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:17.109036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d065ee6-c74d-454d-87e8-0e8f152cc692 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:17.173247 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1026c5-d767-48da-9a8f-9bcec20b63ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:17.174223 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1026c5-d767-48da-9a8f-9bcec20b63ca None admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:17.174772 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1026c5-d767-48da-9a8f-9bcec20b63ca None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:17.175272 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1026c5-d767-48da-9a8f-9bcec20b63ca None admin] PATH_INFO: `/v3/endpoints` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:17.176821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1026c5-d767-48da-9a8f-9bcec20b63ca None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:17.179567 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1026c5-d767-48da-9a8f-9bcec20b63ca None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:17.212296 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde2fd45-cb77-4d48-85ec-326866ec8996 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:17.212944 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde2fd45-cb77-4d48-85ec-326866ec8996 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:17.213476 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde2fd45-cb77-4d48-85ec-326866ec8996 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:17.215360 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 470/937] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:44:17.221081 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fdde15-efed-4de8-bb7c-39ed25031eb3 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:17.221673 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fdde15-efed-4de8-bb7c-39ed25031eb3 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:17.222160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fdde15-efed-4de8-bb7c-39ed25031eb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:17.222423 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 468/938] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:44:16 2022] POST /identity/v3/endpoints => generated 347 bytes in 305 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:44:17.248288 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-f7fdde15-efed-4de8-bb7c-39ed25031eb3 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:17.250210 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-f7fdde15-efed-4de8-bb7c-39ed25031eb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:17.361362 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 471/939] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:17 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 18 00:44:17.384341 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dda3eb2-dd9c-471f-90b4-787890c1fd1c None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:17.384834 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dda3eb2-dd9c-471f-90b4-787890c1fd1c None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:17.385938 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dda3eb2-dd9c-471f-90b4-787890c1fd1c None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:17.389057 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 469/940] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:17 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:17.394945 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21453257-d691-4679-a861-0baa8d053a3a None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:17.395260 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21453257-d691-4679-a861-0baa8d053a3a None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:17.395501 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21453257-d691-4679-a861-0baa8d053a3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:17.402869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:17.417252 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-21453257-d691-4679-a861-0baa8d053a3a None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:17.418547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-21453257-d691-4679-a861-0baa8d053a3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:17.506758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 472/941] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:17 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 18 00:44:17.642067 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dda3eb2-dd9c-471f-90b4-787890c1fd1c None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:17.793635 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21453257-d691-4679-a861-0baa8d053a3a None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:17.794314 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dda3eb2-dd9c-471f-90b4-787890c1fd1c None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:17.814362 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:17.860318 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12d10d1-d85a-419a-8d08-0387e9bd342c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:17.861422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12d10d1-d85a-419a-8d08-0387e9bd342c None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:17.861902 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12d10d1-d85a-419a-8d08-0387e9bd342c None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:17.862386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12d10d1-d85a-419a-8d08-0387e9bd342c None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:17.883095 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12d10d1-d85a-419a-8d08-0387e9bd342c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:44:17.893452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-b12d10d1-d85a-419a-8d08-0387e9bd342c None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 18 00:44:17.893452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:17.893452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:17.893452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:17.893452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:17.893452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:17.893452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:17.893452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:17.893452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:17.893452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:17.893452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:17.893452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:17.893452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:44:17.893452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:44:17.893452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:44:17.893452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:44:17.895372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:44:17.895372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:44:17.895372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:44:17.895372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:44:17.895372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:17.895372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:17.895372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:17.895372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:17.895372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:17.895372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:17.895372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:17.895372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:17.895372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:17.895372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:17.895372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:17.895372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:17.897563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:17.897563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:17.897563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:17.897563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:17.897563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:44:17.897563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:44:17.897563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:44:17.897563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:44:17.897563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:44:17.897563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:44:17.897563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 18 00:44:17.897563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:17.897563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 470/942] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Sep 18 00:44:17 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:17.908382 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12d10d1-d85a-419a-8d08-0387e9bd342c None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:17.978664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21453257-d691-4679-a861-0baa8d053a3a None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:18.038430 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:18.049675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed81a8d5-fdde-482c-907e-e354830c340e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:18.050656 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed81a8d5-fdde-482c-907e-e354830c340e None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:18.051138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed81a8d5-fdde-482c-907e-e354830c340e None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:18.051932 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed81a8d5-fdde-482c-907e-e354830c340e None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:18.053318 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed81a8d5-fdde-482c-907e-e354830c340e None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:18.056309 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed81a8d5-fdde-482c-907e-e354830c340e None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:18.061475 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: WARNING keystone.server.flask.common [None req-ed81a8d5-fdde-482c-907e-e354830c340e None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 18 00:44:18.086686 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12d10d1-d85a-419a-8d08-0387e9bd342c None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:18.098927 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.sql.core [None req-ed81a8d5-fdde-482c-907e-e354830c340e None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 18 00:44:18.098927 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 18 00:44:18.098927 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [parameters: {'id': 'e1980c5ea550429397a647b718f641b5', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 18 00:44:18.098927 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81529) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 18 00:44:18.115049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-ed81a8d5-fdde-482c-907e-e354830c340e None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id e1980c5ea550429397a647b718f641b5.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id e1980c5ea550429397a647b718f641b5. Sep 18 00:44:18.115049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:18.115049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 18 00:44:18.115049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 18 00:44:18.115049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 18 00:44:18.115049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 18 00:44:18.115049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 18 00:44:18.115049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application result = self._query(query) Sep 18 00:44:18.115049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 18 00:44:18.115049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application conn.query(q) Sep 18 00:44:18.115049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 18 00:44:18.115049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 18 00:44:18.115049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 18 00:44:18.115920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application result.read() Sep 18 00:44:18.115920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 18 00:44:18.115920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 18 00:44:18.115920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 18 00:44:18.115920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 18 00:44:18.115920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 18 00:44:18.115920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 18 00:44:18.115920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 18 00:44:18.115920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 18 00:44:18.115920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 18 00:44:18.115920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:18.115920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 18 00:44:18.115920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:18.115920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:18.115920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 18 00:44:18.115920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 18 00:44:18.115920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Sep 18 00:44:18.115920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return project_ref.to_dict() Sep 18 00:44:18.116865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 18 00:44:18.116865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application next(self.gen) Sep 18 00:44:18.116865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 18 00:44:18.116865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application yield resource Sep 18 00:44:18.116865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 18 00:44:18.116865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application next(self.gen) Sep 18 00:44:18.116865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 18 00:44:18.116865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self.session.rollback() Sep 18 00:44:18.116865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 18 00:44:18.116865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self.force_reraise() Sep 18 00:44:18.116865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 18 00:44:18.116865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise self.value Sep 18 00:44:18.116865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 18 00:44:18.116865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 18 00:44:18.116865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 18 00:44:18.116865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application session.commit() Sep 18 00:44:18.116865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 18 00:44:18.117764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 18 00:44:18.117764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 18 00:44:18.117764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._prepare_impl() Sep 18 00:44:18.117764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 18 00:44:18.117764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self.session.flush() Sep 18 00:44:18.117764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 18 00:44:18.117764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._flush(objects) Sep 18 00:44:18.117764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 18 00:44:18.117764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 18 00:44:18.117764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 18 00:44:18.117764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application compat.raise_( Sep 18 00:44:18.117764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 18 00:44:18.117764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception Sep 18 00:44:18.117764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 18 00:44:18.117764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application flush_context.execute() Sep 18 00:44:18.117764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 18 00:44:18.117764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rec.execute(self) Sep 18 00:44:18.118708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 18 00:44:18.118708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 18 00:44:18.118708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 18 00:44:18.118708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 18 00:44:18.118708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 18 00:44:18.118708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 18 00:44:18.118708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 18 00:44:18.118708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 18 00:44:18.118708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 18 00:44:18.118708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 18 00:44:18.118708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 18 00:44:18.118708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 18 00:44:18.118708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 18 00:44:18.118708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 18 00:44:18.118708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 18 00:44:18.119495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 18 00:44:18.119495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 18 00:44:18.119495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception Sep 18 00:44:18.119495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 18 00:44:18.119495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 18 00:44:18.119495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 18 00:44:18.119495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 18 00:44:18.119495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 18 00:44:18.119495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application result = self._query(query) Sep 18 00:44:18.119495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 18 00:44:18.119495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application conn.query(q) Sep 18 00:44:18.119495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 18 00:44:18.119495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 18 00:44:18.119495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 18 00:44:18.119495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application result.read() Sep 18 00:44:18.119495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 18 00:44:18.119495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 18 00:44:18.120405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 18 00:44:18.120405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 18 00:44:18.120405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 18 00:44:18.120405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 18 00:44:18.120405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 18 00:44:18.120405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 18 00:44:18.120405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 18 00:44:18.120405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 18 00:44:18.120405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [parameters: {'id': 'e1980c5ea550429397a647b718f641b5', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 18 00:44:18.120405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 18 00:44:18.120405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:18.120405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:44:18.120405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:18.120405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:18.120405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Sep 18 00:44:18.121180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Sep 18 00:44:18.121180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 18 00:44:18.121180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 18 00:44:18.121180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Sep 18 00:44:18.121180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:18.121180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:44:18.121180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:18.121180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:18.121180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:18.121180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:18.121180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:18.121180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:18.121180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:18.121180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:18.121180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:18.121180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:18.121180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:18.122101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:18.122101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Sep 18 00:44:18.122101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Sep 18 00:44:18.122101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:18.122101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:18.122101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Sep 18 00:44:18.122101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.Conflict( Sep 18 00:44:18.122101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id e1980c5ea550429397a647b718f641b5. Sep 18 00:44:18.122101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:18.127942 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 473/943] 10.222.0.42 () {60 vars in 1140 bytes} [Sun Sep 18 00:44:17 2022] POST /identity/v3/projects => generated 262 bytes in 326 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:18.130079 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed81a8d5-fdde-482c-907e-e354830c340e None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:18.152872 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07ac569-9968-4938-90e3-08e6bb08381b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:18.154019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07ac569-9968-4938-90e3-08e6bb08381b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:18.154687 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07ac569-9968-4938-90e3-08e6bb08381b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:18.155344 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07ac569-9968-4938-90e3-08e6bb08381b None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:18.156656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07ac569-9968-4938-90e3-08e6bb08381b None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:18.157345 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07ac569-9968-4938-90e3-08e6bb08381b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:18.160135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07ac569-9968-4938-90e3-08e6bb08381b None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:18.176733 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 471/944] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Sep 18 00:44:17 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:18.178495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8e02ec-e001-4989-8dab-c55ab6d98ffd None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:18.178881 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8e02ec-e001-4989-8dab-c55ab6d98ffd None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:18.179442 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8e02ec-e001-4989-8dab-c55ab6d98ffd None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:18.181760 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 472/945] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:18 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:18.187688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8e02ec-e001-4989-8dab-c55ab6d98ffd None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:18.241086 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:18.311401 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed81a8d5-fdde-482c-907e-e354830c340e None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:18.352695 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8e02ec-e001-4989-8dab-c55ab6d98ffd None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:18.355278 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1fe679-8901-4d31-b69d-aff4676b5258 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:18.355947 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1fe679-8901-4d31-b69d-aff4676b5258 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:18.356229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1fe679-8901-4d31-b69d-aff4676b5258 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:18.356501 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1fe679-8901-4d31-b69d-aff4676b5258 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:18.364544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1fe679-8901-4d31-b69d-aff4676b5258 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:18.366258 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1fe679-8901-4d31-b69d-aff4676b5258 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:18.382198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-ca1fe679-8901-4d31-b69d-aff4676b5258 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:44:18.382198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:18.382198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:18.382198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:18.382198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:18.382198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:18.382198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:18.382198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:18.382198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:18.382198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:18.382198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:18.382198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:18.382198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:44:18.382198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:44:18.382198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:44:18.383977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:44:18.383977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:18.383977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:18.383977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:18.383977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:18.383977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:18.383977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:18.383977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:18.383977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:18.383977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:18.383977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:18.383977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:18.383977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:18.383977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:18.383977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:18.383977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:18.383977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:18.385787 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:44:18.385787 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:44:18.385787 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:44:18.385787 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:44:18.385787 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:44:18.385787 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:44:18.385787 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:44:18.385787 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:18.385787 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 474/946] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:44:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:18.386814 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de01ba1-2f7d-484c-91ed-190fe5e61335 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:18.387852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de01ba1-2f7d-484c-91ed-190fe5e61335 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:18.387852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de01ba1-2f7d-484c-91ed-190fe5e61335 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:18.404688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532b11ef-344c-4444-9e8f-912e093dcaa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:18.405341 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532b11ef-344c-4444-9e8f-912e093dcaa6 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:18.405623 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532b11ef-344c-4444-9e8f-912e093dcaa6 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:18.405898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532b11ef-344c-4444-9e8f-912e093dcaa6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:18.412458 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-0de01ba1-2f7d-484c-91ed-190fe5e61335 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:18.414447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-0de01ba1-2f7d-484c-91ed-190fe5e61335 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:18.420699 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-532b11ef-344c-4444-9e8f-912e093dcaa6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:18.422734 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-532b11ef-344c-4444-9e8f-912e093dcaa6 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:18.440678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-532b11ef-344c-4444-9e8f-912e093dcaa6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:18.440678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:18.440678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:18.440678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:18.440678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:18.440678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:18.440678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:18.440678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:18.440678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:18.440678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:18.440678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:18.440678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:18.440678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:44:18.440678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:44:18.440678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:44:18.440678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:44:18.442885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:18.442885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:18.442885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:18.442885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:18.442885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:18.442885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:18.442885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:18.442885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:18.442885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:18.442885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:18.442885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:18.442885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:18.442885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:18.442885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:18.442885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:18.442885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:18.442885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:44:18.444677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:44:18.444677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:18.444677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:18.444677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 473/947] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:18.451366 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532b11ef-344c-4444-9e8f-912e093dcaa6 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:18.483309 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:18.526786 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 475/948] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:18 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:18.526786 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de01ba1-2f7d-484c-91ed-190fe5e61335 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:18.597413 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:18.631175 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532b11ef-344c-4444-9e8f-912e093dcaa6 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:18.690071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de01ba1-2f7d-484c-91ed-190fe5e61335 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:18.705441 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ca8cb0-316d-4a95-9a41-82541116f153 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:18.707382 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ca8cb0-316d-4a95-9a41-82541116f153 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:18.707810 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ca8cb0-316d-4a95-9a41-82541116f153 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:18.708222 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ca8cb0-316d-4a95-9a41-82541116f153 None admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:18.709034 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ca8cb0-316d-4a95-9a41-82541116f153 None admin] RBAC: Adding query filter params (name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:18.709462 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ca8cb0-316d-4a95-9a41-82541116f153 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:18.711488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ca8cb0-316d-4a95-9a41-82541116f153 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:18.750492 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 474/949] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 18 00:44:18 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:18.767929 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044826ed-4c4a-44dd-8bd3-19f7404e4054 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:18.768900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044826ed-4c4a-44dd-8bd3-19f7404e4054 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:18.769412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044826ed-4c4a-44dd-8bd3-19f7404e4054 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:18.769868 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044826ed-4c4a-44dd-8bd3-19f7404e4054 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:18.771025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044826ed-4c4a-44dd-8bd3-19f7404e4054 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:18.771581 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044826ed-4c4a-44dd-8bd3-19f7404e4054 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:18.774003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044826ed-4c4a-44dd-8bd3-19f7404e4054 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:18.800030 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 476/950] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:44:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:18.809409 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ca8cb0-316d-4a95-9a41-82541116f153 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:18.877584 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044826ed-4c4a-44dd-8bd3-19f7404e4054 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:18.950349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:19.001242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ca8cb0-316d-4a95-9a41-82541116f153 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:19.065186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044826ed-4c4a-44dd-8bd3-19f7404e4054 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:19.073791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188798fd-a8d9-471a-a7f4-4abc646b4f9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:19.074885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188798fd-a8d9-471a-a7f4-4abc646b4f9a None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:19.075348 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188798fd-a8d9-471a-a7f4-4abc646b4f9a None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:19.075825 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188798fd-a8d9-471a-a7f4-4abc646b4f9a None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:19.096877 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188798fd-a8d9-471a-a7f4-4abc646b4f9a None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:19.099832 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188798fd-a8d9-471a-a7f4-4abc646b4f9a None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:19.118738 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e789b7-db1f-4b49-a0b5-88e974e1955d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:19.120091 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e789b7-db1f-4b49-a0b5-88e974e1955d None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:19.120284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e789b7-db1f-4b49-a0b5-88e974e1955d None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:19.120642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e789b7-db1f-4b49-a0b5-88e974e1955d None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:19.121327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e789b7-db1f-4b49-a0b5-88e974e1955d None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:19.123103 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e789b7-db1f-4b49-a0b5-88e974e1955d None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:19.126866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-188798fd-a8d9-471a-a7f4-4abc646b4f9a None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 18 00:44:19.126866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:19.126866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:19.126866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:19.126866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:19.126866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:19.126866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:19.126866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:19.126866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:19.126866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:19.126866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:19.126866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:19.126866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:44:19.126866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:44:19.126866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:44:19.126866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:44:19.128819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:19.128819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:19.128819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:44:19.128819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:44:19.128819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:44:19.128819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:44:19.128819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:19.128819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:19.128819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:19.128819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:19.128819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:19.128819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:19.128819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:19.128819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:19.128819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:19.128819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:19.128819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:19.128819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:19.130040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:19.130040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:19.130040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:44:19.130040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:44:19.130040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:44:19.130040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:44:19.130040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:44:19.130040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:44:19.130040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 18 00:44:19.130040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:19.130040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 475/951] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Sep 18 00:44:18 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 321 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:19.139454 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188798fd-a8d9-471a-a7f4-4abc646b4f9a None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:19.144715 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:19.151578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-28e789b7-db1f-4b49-a0b5-88e974e1955d None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 18 00:44:19.151578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:19.151578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:44:19.151578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:44:19.151578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:44:19.151578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:44:19.151578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:44:19.151578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:19.151578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 18 00:44:19.151578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:19.151578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:44:19.151578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:19.151578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:19.151578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:19.151578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:19.151578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:19.151578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:19.154825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:19.154825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:19.154825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:19.154825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:19.154825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:19.154825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:19.154825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:44:19.154825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:44:19.154825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:44:19.154825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:44:19.154825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:19.154825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:19.154825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:19.154825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:19.154825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:19.154825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:19.156723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:19.156723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:19.156723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:19.156723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:19.156723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:19.156723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:19.156723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:19.156723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:19.156723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:19.156723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:19.156723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:44:19.156723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:19.156723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 18 00:44:19.156723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:19.156723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 477/952] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Sep 18 00:44:18 2022] GET /identity/v3/services/object-store => generated 93 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:19.161378 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e789b7-db1f-4b49-a0b5-88e974e1955d None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:19.331667 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188798fd-a8d9-471a-a7f4-4abc646b4f9a None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:19.344889 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e789b7-db1f-4b49-a0b5-88e974e1955d None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:19.348999 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:19.390446 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3d2a7a-35e8-4ac9-bc36-a845d0a8c6ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:19.390994 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3d2a7a-35e8-4ac9-bc36-a845d0a8c6ed None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:19.391306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3d2a7a-35e8-4ac9-bc36-a845d0a8c6ed None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:19.391581 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3d2a7a-35e8-4ac9-bc36-a845d0a8c6ed None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:19.392191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3d2a7a-35e8-4ac9-bc36-a845d0a8c6ed None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:19.392523 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3d2a7a-35e8-4ac9-bc36-a845d0a8c6ed None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:19.394120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3d2a7a-35e8-4ac9-bc36-a845d0a8c6ed None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:19.400782 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 478/953] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Sep 18 00:44:19 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:19.408107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3d2a7a-35e8-4ac9-bc36-a845d0a8c6ed None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:19.422495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb15635c-7359-4ee7-9673-ff28b11145bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:19.422495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb15635c-7359-4ee7-9673-ff28b11145bd None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:19.422495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb15635c-7359-4ee7-9673-ff28b11145bd None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:19.422495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb15635c-7359-4ee7-9673-ff28b11145bd None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:19.423138 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb15635c-7359-4ee7-9673-ff28b11145bd None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:19.423138 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb15635c-7359-4ee7-9673-ff28b11145bd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:19.424766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb15635c-7359-4ee7-9673-ff28b11145bd None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:19.452942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 476/954] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Sep 18 00:44:19 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:19.459805 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb15635c-7359-4ee7-9673-ff28b11145bd None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:19.546033 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:19.556918 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3d2a7a-35e8-4ac9-bc36-a845d0a8c6ed None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:19.595451 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb15635c-7359-4ee7-9673-ff28b11145bd None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:19.597362 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753d4cd0-b7ad-4922-bce4-bcc4e49b753a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:19.598066 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753d4cd0-b7ad-4922-bce4-bcc4e49b753a None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:19.598411 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753d4cd0-b7ad-4922-bce4-bcc4e49b753a None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:19.598803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753d4cd0-b7ad-4922-bce4-bcc4e49b753a None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:19.600202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753d4cd0-b7ad-4922-bce4-bcc4e49b753a None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:19.600202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753d4cd0-b7ad-4922-bce4-bcc4e49b753a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:19.601297 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753d4cd0-b7ad-4922-bce4-bcc4e49b753a None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:19.612629 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 479/955] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Sep 18 00:44:19 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:19.613767 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753d4cd0-b7ad-4922-bce4-bcc4e49b753a None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:19.637867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd95c65-e3d9-492e-b249-5823b51c6372 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:19.638891 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd95c65-e3d9-492e-b249-5823b51c6372 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:19.639381 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd95c65-e3d9-492e-b249-5823b51c6372 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:19.639885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd95c65-e3d9-492e-b249-5823b51c6372 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:19.647861 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:19.720471 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd95c65-e3d9-492e-b249-5823b51c6372 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:19.722791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd95c65-e3d9-492e-b249-5823b51c6372 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:19.726070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753d4cd0-b7ad-4922-bce4-bcc4e49b753a None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:19.769495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5464cd1e-62ba-410e-991b-7f6569838872 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:19.770115 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5464cd1e-62ba-410e-991b-7f6569838872 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:19.770395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5464cd1e-62ba-410e-991b-7f6569838872 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:19.770691 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5464cd1e-62ba-410e-991b-7f6569838872 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:19.771405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5464cd1e-62ba-410e-991b-7f6569838872 None admin] RBAC: Adding query filter params (interface=public, service_id=8af681d8c9ba478dba73dac151cb8b0c, region_id=RegionOne) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:19.771709 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5464cd1e-62ba-410e-991b-7f6569838872 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:19.773081 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5464cd1e-62ba-410e-991b-7f6569838872 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:19.778798 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 480/956] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Sep 18 00:44:19 2022] GET /identity/v3/endpoints?service_id=8af681d8c9ba478dba73dac151cb8b0c&interface=public®ion_id=RegionOne => generated 194 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:19.780072 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36beca49-800a-4ec1-b15c-fda7c9818a56 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:19.780354 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36beca49-800a-4ec1-b15c-fda7c9818a56 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:19.780623 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36beca49-800a-4ec1-b15c-fda7c9818a56 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:19.783645 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 481/957] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:19.784299 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36beca49-800a-4ec1-b15c-fda7c9818a56 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:19.812729 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:19.855610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 477/958] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:44:19 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 18 00:44:19.857022 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3cba4f-7bcd-4ed7-8626-0182d3bd5378 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:19.857403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3cba4f-7bcd-4ed7-8626-0182d3bd5378 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:19.857752 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3cba4f-7bcd-4ed7-8626-0182d3bd5378 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:19.877432 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-dc3cba4f-7bcd-4ed7-8626-0182d3bd5378 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:19.878698 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-dc3cba4f-7bcd-4ed7-8626-0182d3bd5378 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:19.900237 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:19.914475 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36beca49-800a-4ec1-b15c-fda7c9818a56 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:19.964941 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 478/959] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:19 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:19.965813 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3cba4f-7bcd-4ed7-8626-0182d3bd5378 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:19.975634 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc31780d-fb4a-4a34-8dd3-a9c39e1902e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:19.976492 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc31780d-fb4a-4a34-8dd3-a9c39e1902e7 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:19.976922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc31780d-fb4a-4a34-8dd3-a9c39e1902e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:19.977468 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc31780d-fb4a-4a34-8dd3-a9c39e1902e7 None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:19.978236 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc31780d-fb4a-4a34-8dd3-a9c39e1902e7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:19.980028 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc31780d-fb4a-4a34-8dd3-a9c39e1902e7 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:19.987416 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 482/960] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Sep 18 00:44:19 2022] GET /identity/v3/services => generated 1497 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:44:20.148185 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:20.159604 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3cba4f-7bcd-4ed7-8626-0182d3bd5378 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:20.207206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67532d89-da1b-47e6-add7-8ac5207447ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:20.208255 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67532d89-da1b-47e6-add7-8ac5207447ac None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:20.208747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67532d89-da1b-47e6-add7-8ac5207447ac None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:20.209426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67532d89-da1b-47e6-add7-8ac5207447ac None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:20.223818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67532d89-da1b-47e6-add7-8ac5207447ac None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:20.225810 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67532d89-da1b-47e6-add7-8ac5207447ac None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:20.244149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-67532d89-da1b-47e6-add7-8ac5207447ac None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:20.244149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:20.244149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:20.244149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:20.244149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:20.244149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:20.244149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:20.244149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:20.244149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:20.244149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:20.244149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:20.244149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:20.244149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:44:20.244149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:44:20.244149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:44:20.244149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:44:20.245780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:20.245780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:20.245780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:20.245780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:20.245780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:20.245780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:20.245780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:20.245780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:20.245780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:20.245780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:20.245780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:20.245780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:20.245780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:20.245780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:20.245780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:20.245780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:20.245780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:44:20.246769 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:44:20.246769 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:20.246769 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:20.248847 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 479/961] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:20.257620 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc31780d-fb4a-4a34-8dd3-a9c39e1902e7 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:20.259408 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67532d89-da1b-47e6-add7-8ac5207447ac None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:20.387118 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc31780d-fb4a-4a34-8dd3-a9c39e1902e7 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:20.404490 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67532d89-da1b-47e6-add7-8ac5207447ac None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:20.423490 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9161f3dd-617f-4eaa-ae11-e9b6c758a24d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:20.424038 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9161f3dd-617f-4eaa-ae11-e9b6c758a24d None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:20.424316 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9161f3dd-617f-4eaa-ae11-e9b6c758a24d None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:20.424573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9161f3dd-617f-4eaa-ae11-e9b6c758a24d None admin] PATH_INFO: `/v3/services/compute` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:20.425228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9161f3dd-617f-4eaa-ae11-e9b6c758a24d None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:20.426914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9161f3dd-617f-4eaa-ae11-e9b6c758a24d None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:20.436188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-9161f3dd-617f-4eaa-ae11-e9b6c758a24d None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 18 00:44:20.436188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:20.436188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:44:20.436188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:44:20.436188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:44:20.436188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:44:20.436188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:44:20.436188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:20.436188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 18 00:44:20.436188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:20.436188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:44:20.436188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:20.436188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:20.436188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:20.436188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:20.436188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:20.436188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:20.438333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:20.438333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:20.438333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:20.438333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:20.438333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:20.438333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:20.438333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:44:20.438333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:44:20.438333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:44:20.438333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:44:20.438333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:20.438333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:20.438333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:20.438333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:20.438333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:20.438333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:20.440034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:20.440034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:20.440034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:20.440034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:20.440034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:20.440034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:20.440034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:20.440034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:20.440034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:20.440034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:20.440034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:44:20.440034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:20.440034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 18 00:44:20.440034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:20.440034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 483/962] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:44:20 2022] GET /identity/v3/services/compute => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:20.446617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9161f3dd-617f-4eaa-ae11-e9b6c758a24d None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:20.453221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262d1e4f-be61-4a3c-9f41-4c6f0f5aa015 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:20.453221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262d1e4f-be61-4a3c-9f41-4c6f0f5aa015 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:20.453481 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262d1e4f-be61-4a3c-9f41-4c6f0f5aa015 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:20.453705 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262d1e4f-be61-4a3c-9f41-4c6f0f5aa015 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:20.454358 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-262d1e4f-be61-4a3c-9f41-4c6f0f5aa015 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:20.454740 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-262d1e4f-be61-4a3c-9f41-4c6f0f5aa015 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:20.456104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-262d1e4f-be61-4a3c-9f41-4c6f0f5aa015 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:20.469871 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 480/963] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:44:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:20.477827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262d1e4f-be61-4a3c-9f41-4c6f0f5aa015 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:20.482798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:20.489751 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:20.610574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9161f3dd-617f-4eaa-ae11-e9b6c758a24d None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:20.620562 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262d1e4f-be61-4a3c-9f41-4c6f0f5aa015 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:20.665804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2a7f3a-9048-47c2-a3fe-33abf4115384 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:20.666107 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084389c7-c984-4eca-96b8-ad424719ec76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:20.666682 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084389c7-c984-4eca-96b8-ad424719ec76 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:20.666964 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084389c7-c984-4eca-96b8-ad424719ec76 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:20.667248 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084389c7-c984-4eca-96b8-ad424719ec76 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:20.668497 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2a7f3a-9048-47c2-a3fe-33abf4115384 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:20.669280 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2a7f3a-9048-47c2-a3fe-33abf4115384 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:20.669963 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2a7f3a-9048-47c2-a3fe-33abf4115384 None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:20.671368 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2a7f3a-9048-47c2-a3fe-33abf4115384 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:20.672271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2a7f3a-9048-47c2-a3fe-33abf4115384 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:20.674649 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084389c7-c984-4eca-96b8-ad424719ec76 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:20.675743 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2a7f3a-9048-47c2-a3fe-33abf4115384 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:20.676010 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084389c7-c984-4eca-96b8-ad424719ec76 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:20.686672 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 484/964] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 18 00:44:20 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:20.693827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-084389c7-c984-4eca-96b8-ad424719ec76 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:44:20.693827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:20.693827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:20.693827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:20.693827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:20.693827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:20.693827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:20.693827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:20.693827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:20.693827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:20.693827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:20.693827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:20.693827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:44:20.693827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:44:20.693827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:44:20.695584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:44:20.695584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:20.695584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:20.695584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:20.695584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:20.695584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:20.695584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:20.695584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:20.695584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:20.695584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:20.695584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:20.695584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:20.695584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:20.695584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:20.695584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:20.695584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:20.695584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:20.697531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:44:20.697531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:44:20.697531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:44:20.697531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:44:20.697531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:44:20.697531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:44:20.697531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:44:20.697531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:20.699106 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2a7f3a-9048-47c2-a3fe-33abf4115384 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:20.700620 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 481/965] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:44:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:20.706963 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084389c7-c984-4eca-96b8-ad424719ec76 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:20.802844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:20.828404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:20.848702 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2a7f3a-9048-47c2-a3fe-33abf4115384 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:20.859886 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084389c7-c984-4eca-96b8-ad424719ec76 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:20.909083 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b03354c-afcc-46a6-b209-0acec7588027 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:20.909765 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda95c7b-7df2-4df4-a8cb-ab10916f865e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:20.910339 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda95c7b-7df2-4df4-a8cb-ab10916f865e None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:20.910658 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda95c7b-7df2-4df4-a8cb-ab10916f865e None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:20.910947 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda95c7b-7df2-4df4-a8cb-ab10916f865e None admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:20.911327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b03354c-afcc-46a6-b209-0acec7588027 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:20.911625 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda95c7b-7df2-4df4-a8cb-ab10916f865e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:20.911930 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda95c7b-7df2-4df4-a8cb-ab10916f865e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:20.912211 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b03354c-afcc-46a6-b209-0acec7588027 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:20.912684 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b03354c-afcc-46a6-b209-0acec7588027 None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:20.913364 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda95c7b-7df2-4df4-a8cb-ab10916f865e None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:20.913737 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b03354c-afcc-46a6-b209-0acec7588027 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:20.914236 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b03354c-afcc-46a6-b209-0acec7588027 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:20.917408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b03354c-afcc-46a6-b209-0acec7588027 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:20.923915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 485/966] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 18 00:44:20 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:20.940334 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 482/967] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:44:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:20.942332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b03354c-afcc-46a6-b209-0acec7588027 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:21.071532 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b03354c-afcc-46a6-b209-0acec7588027 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:21.112979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4635ec98-e5e4-49a5-aa1d-8e201672a3c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:21.113611 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4635ec98-e5e4-49a5-aa1d-8e201672a3c2 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:21.113894 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4635ec98-e5e4-49a5-aa1d-8e201672a3c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:21.114170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4635ec98-e5e4-49a5-aa1d-8e201672a3c2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:21.114966 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4635ec98-e5e4-49a5-aa1d-8e201672a3c2 None admin] RBAC: Adding query filter params (role.id=88c42ae856ff4e2d8a42a5b1d50b97f9, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=3e31dce69c344c9aa4a55f31b9c735df) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:21.115283 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4635ec98-e5e4-49a5-aa1d-8e201672a3c2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:21.116757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4635ec98-e5e4-49a5-aa1d-8e201672a3c2 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:21.125335 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 486/968] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:44:20 2022] GET /identity/v3/role_assignments?user.id=3e31dce69c344c9aa4a55f31b9c735df&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=88c42ae856ff4e2d8a42a5b1d50b97f9 => generated 608 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:21.330286 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7983773b-db65-41be-a97b-225322a8942d None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:21.330810 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7983773b-db65-41be-a97b-225322a8942d None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:21.330913 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7983773b-db65-41be-a97b-225322a8942d None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:21.332853 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 483/969] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:21.339438 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070c8535-7e2d-46ef-aaa6-09fa72a29888 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:21.340028 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070c8535-7e2d-46ef-aaa6-09fa72a29888 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:21.340549 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070c8535-7e2d-46ef-aaa6-09fa72a29888 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:21.360899 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-070c8535-7e2d-46ef-aaa6-09fa72a29888 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:21.362493 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-070c8535-7e2d-46ef-aaa6-09fa72a29888 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:21.374316 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:21.442594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 487/970] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:21 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:21.714524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7983773b-db65-41be-a97b-225322a8942d None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:21.857393 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7983773b-db65-41be-a97b-225322a8942d None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:21.887140 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:21.900093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9811a86-2c63-43b4-a7a5-493b8f61b9a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:21.900715 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9811a86-2c63-43b4-a7a5-493b8f61b9a5 None admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:21.901573 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9811a86-2c63-43b4-a7a5-493b8f61b9a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:21.901845 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9811a86-2c63-43b4-a7a5-493b8f61b9a5 None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:21.902550 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9811a86-2c63-43b4-a7a5-493b8f61b9a5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:21.904018 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9811a86-2c63-43b4-a7a5-493b8f61b9a5 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:21.906728 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: WARNING keystone.server.flask.common [None req-d9811a86-2c63-43b4-a7a5-493b8f61b9a5 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 18 00:44:21.920583 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.sql.core [None req-d9811a86-2c63-43b4-a7a5-493b8f61b9a5 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 18 00:44:21.920583 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 18 00:44:21.920583 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [parameters: {'user_id': '7d98cd45b88b4b8ca5a3e8eeaef79ed7', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 18 00:44:21.920583 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81530) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 18 00:44:21.939453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-d9811a86-2c63-43b4-a7a5-493b8f61b9a5 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 18 00:44:21.939453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:21.939453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 18 00:44:21.939453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 18 00:44:21.939453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 18 00:44:21.939453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 18 00:44:21.939453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 18 00:44:21.939453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application result = self._query(query) Sep 18 00:44:21.939453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 18 00:44:21.939453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application conn.query(q) Sep 18 00:44:21.939453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 18 00:44:21.939453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 18 00:44:21.939453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 18 00:44:21.939453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application result.read() Sep 18 00:44:21.939453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 18 00:44:21.940426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 18 00:44:21.940426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 18 00:44:21.940426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 18 00:44:21.940426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 18 00:44:21.940426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 18 00:44:21.940426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 18 00:44:21.940426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 18 00:44:21.940426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 18 00:44:21.940426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:21.940426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 18 00:44:21.940426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:21.940426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:21.940426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 18 00:44:21.940426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 18 00:44:21.940426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Sep 18 00:44:21.940426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Sep 18 00:44:21.940426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 18 00:44:21.940426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application next(self.gen) Sep 18 00:44:21.941447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 18 00:44:21.941447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application yield resource Sep 18 00:44:21.941447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 18 00:44:21.941447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application next(self.gen) Sep 18 00:44:21.941447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 18 00:44:21.941447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self.session.rollback() Sep 18 00:44:21.941447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 18 00:44:21.941447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self.force_reraise() Sep 18 00:44:21.941447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 18 00:44:21.941447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise self.value Sep 18 00:44:21.941447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 18 00:44:21.941447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 18 00:44:21.941447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 18 00:44:21.941447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application session.commit() Sep 18 00:44:21.941447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 18 00:44:21.941447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 18 00:44:21.941447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 18 00:44:21.942361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self._prepare_impl() Sep 18 00:44:21.942361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 18 00:44:21.942361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self.session.flush() Sep 18 00:44:21.942361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 18 00:44:21.942361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self._flush(objects) Sep 18 00:44:21.942361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 18 00:44:21.942361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 18 00:44:21.942361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 18 00:44:21.942361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application compat.raise_( Sep 18 00:44:21.942361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 18 00:44:21.942361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception Sep 18 00:44:21.942361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 18 00:44:21.942361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application flush_context.execute() Sep 18 00:44:21.942361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 18 00:44:21.942361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rec.execute(self) Sep 18 00:44:21.942361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 18 00:44:21.942361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 18 00:44:21.943304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 18 00:44:21.943304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 18 00:44:21.943304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Sep 18 00:44:21.943304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application result = connection._execute_20( Sep 18 00:44:21.943304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 18 00:44:21.943304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 18 00:44:21.943304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 18 00:44:21.943304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 18 00:44:21.943304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 18 00:44:21.943304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 18 00:44:21.943304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 18 00:44:21.943304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 18 00:44:21.943304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 18 00:44:21.943304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 18 00:44:21.943304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 18 00:44:21.943304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception Sep 18 00:44:21.944170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 18 00:44:21.944170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 18 00:44:21.944170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 18 00:44:21.944170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 18 00:44:21.944170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 18 00:44:21.944170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application result = self._query(query) Sep 18 00:44:21.944170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 18 00:44:21.944170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application conn.query(q) Sep 18 00:44:21.944170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 18 00:44:21.944170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 18 00:44:21.944170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 18 00:44:21.944170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application result.read() Sep 18 00:44:21.944170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 18 00:44:21.944170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 18 00:44:21.944170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 18 00:44:21.944170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 18 00:44:21.945037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 18 00:44:21.945037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 18 00:44:21.945037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 18 00:44:21.945037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 18 00:44:21.945037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 18 00:44:21.945037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 18 00:44:21.945037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [parameters: {'user_id': '7d98cd45b88b4b8ca5a3e8eeaef79ed7', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 18 00:44:21.945037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 18 00:44:21.945037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:21.945037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:44:21.945037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:21.945037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:21.945037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:21.945037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:21.945037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:21.945861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:21.945861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:21.945861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:21.945861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:21.945861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:21.945861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:21.945861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:21.945861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Sep 18 00:44:21.945861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Sep 18 00:44:21.945861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:21.945861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:21.945861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:44:21.945861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:44:21.945861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:44:21.945861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:44:21.945861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Sep 18 00:44:21.945861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Sep 18 00:44:21.952190 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Sep 18 00:44:21.952190 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Sep 18 00:44:21.952190 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 18 00:44:21.952190 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 18 00:44:21.952190 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 18 00:44:21.952190 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:21.952190 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 484/971] 10.222.0.42 () {60 vars in 1131 bytes} [Sun Sep 18 00:44:21 2022] POST /identity/v3/users => generated 160 bytes in 227 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:21.952643 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070c8535-7e2d-46ef-aaa6-09fa72a29888 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:22.067733 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070c8535-7e2d-46ef-aaa6-09fa72a29888 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:22.108133 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59b0769-3d63-4030-8718-47a76ff4fe5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:22.109030 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59b0769-3d63-4030-8718-47a76ff4fe5c None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:22.109489 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59b0769-3d63-4030-8718-47a76ff4fe5c None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:22.109914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59b0769-3d63-4030-8718-47a76ff4fe5c None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:22.117963 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59b0769-3d63-4030-8718-47a76ff4fe5c None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:22.120239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59b0769-3d63-4030-8718-47a76ff4fe5c None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:22.133136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-f59b0769-3d63-4030-8718-47a76ff4fe5c None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 18 00:44:22.133136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:22.133136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:22.133136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:22.133136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:22.133136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:22.133136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:22.133136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:22.133136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:22.133136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:22.133136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:22.133136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:22.133136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:44:22.133136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:44:22.133136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:44:22.133136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:44:22.134370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:22.134370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:22.134370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:44:22.134370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:44:22.134370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:44:22.134370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:44:22.134370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:22.134370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:22.134370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:22.134370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:22.134370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:22.134370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:22.134370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:22.134370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:22.134370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:22.134370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:22.134370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:22.134370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:22.135659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:22.135659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:22.135659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:44:22.135659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:44:22.135659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:44:22.135659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:44:22.135659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:44:22.135659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:44:22.135659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 18 00:44:22.135659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:22.137860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 488/972] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:44:21 2022] GET /identity/v3/users/cinder => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:22.144193 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9811a86-2c63-43b4-a7a5-493b8f61b9a5 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:22.269526 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9811a86-2c63-43b4-a7a5-493b8f61b9a5 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:22.306432 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0119642-579c-47ec-86ae-1ded765b96ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:22.307908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0119642-579c-47ec-86ae-1ded765b96ed None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:22.308347 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0119642-579c-47ec-86ae-1ded765b96ed None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:22.308771 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0119642-579c-47ec-86ae-1ded765b96ed None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:22.309634 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0119642-579c-47ec-86ae-1ded765b96ed None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:22.310089 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0119642-579c-47ec-86ae-1ded765b96ed None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:22.312070 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0119642-579c-47ec-86ae-1ded765b96ed None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:22.353985 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 485/973] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 18 00:44:22 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:22.670616 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e1966c-c77a-4776-b2af-868a91c202ee None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:22.670807 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e1966c-c77a-4776-b2af-868a91c202ee None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:22.671086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e1966c-c77a-4776-b2af-868a91c202ee None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:22.672442 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 489/974] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:22.678150 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d0d9fb-ebf5-4a4f-9bb7-5381527dc81d None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:22.678434 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d0d9fb-ebf5-4a4f-9bb7-5381527dc81d None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:22.678736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d0d9fb-ebf5-4a4f-9bb7-5381527dc81d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:22.697147 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-96d0d9fb-ebf5-4a4f-9bb7-5381527dc81d None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:22.699232 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-96d0d9fb-ebf5-4a4f-9bb7-5381527dc81d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:22.745509 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:22.775267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 486/975] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:22 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:23.081671 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e1966c-c77a-4776-b2af-868a91c202ee None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:23.085939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:23.210544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e1966c-c77a-4776-b2af-868a91c202ee None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:23.257389 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26760210-2cb7-41fa-8cd6-632e24b53515 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:23.258055 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26760210-2cb7-41fa-8cd6-632e24b53515 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:23.258438 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26760210-2cb7-41fa-8cd6-632e24b53515 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:23.258938 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26760210-2cb7-41fa-8cd6-632e24b53515 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:23.260060 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26760210-2cb7-41fa-8cd6-632e24b53515 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:23.262636 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26760210-2cb7-41fa-8cd6-632e24b53515 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:23.287624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-26760210-2cb7-41fa-8cd6-632e24b53515 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 18 00:44:23.287624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:23.287624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:44:23.287624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:44:23.287624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:44:23.287624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:44:23.287624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:44:23.287624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:23.287624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 18 00:44:23.287624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:23.287624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:44:23.287624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:23.287624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:23.287624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:23.287624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:23.287624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:23.287624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:23.289434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:23.289434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:23.289434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:23.289434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:23.289434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:23.289434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:23.289434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:44:23.289434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:44:23.289434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:44:23.289434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:44:23.289434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:23.289434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:23.289434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:23.289434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:23.289434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:23.289434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:23.291004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:23.291004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:23.291004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:23.291004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:23.291004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:23.291004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:23.291004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:23.291004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:23.291004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:23.291004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:23.291004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:44:23.291004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:23.291004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 18 00:44:23.291004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:23.295469 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 490/976] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Sep 18 00:44:23 2022] GET /identity/v3/services/object-store => generated 93 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:23.298979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d0d9fb-ebf5-4a4f-9bb7-5381527dc81d None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:23.432881 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d0d9fb-ebf5-4a4f-9bb7-5381527dc81d None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:23.477332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519671f9-8588-4255-83a8-de9aaacecec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:23.478254 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519671f9-8588-4255-83a8-de9aaacecec3 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:23.478706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519671f9-8588-4255-83a8-de9aaacecec3 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:23.479718 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519671f9-8588-4255-83a8-de9aaacecec3 None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:23.480729 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519671f9-8588-4255-83a8-de9aaacecec3 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:23.481187 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519671f9-8588-4255-83a8-de9aaacecec3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:23.483521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519671f9-8588-4255-83a8-de9aaacecec3 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:23.490616 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 487/977] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Sep 18 00:44:23 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:23.497196 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26760210-2cb7-41fa-8cd6-632e24b53515 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:23.564677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224075b5-72fc-4ace-ad23-9b294536bb6f None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:23.565036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224075b5-72fc-4ace-ad23-9b294536bb6f None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:23.565226 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224075b5-72fc-4ace-ad23-9b294536bb6f None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:23.566885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 488/978] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:44:23.575394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f81318-1316-42eb-8819-072e95523272 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:23.575824 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f81318-1316-42eb-8819-072e95523272 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:23.576273 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f81318-1316-42eb-8819-072e95523272 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:23.604377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:23.605385 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-e7f81318-1316-42eb-8819-072e95523272 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:23.607490 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-e7f81318-1316-42eb-8819-072e95523272 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:23.642856 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26760210-2cb7-41fa-8cd6-632e24b53515 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:23.649786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:23.689753 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0983a3-7e78-40d9-8c40-efcd7a07314b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:23.690257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0983a3-7e78-40d9-8c40-efcd7a07314b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:23.690589 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0983a3-7e78-40d9-8c40-efcd7a07314b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:23.690851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0983a3-7e78-40d9-8c40-efcd7a07314b None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:23.691971 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0983a3-7e78-40d9-8c40-efcd7a07314b None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:23.692565 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0983a3-7e78-40d9-8c40-efcd7a07314b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:23.694622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0983a3-7e78-40d9-8c40-efcd7a07314b None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:23.701911 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 491/979] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Sep 18 00:44:23 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:23.703259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00bec2a-aa9b-412e-9f36-4660778c66f2 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:23.703545 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00bec2a-aa9b-412e-9f36-4660778c66f2 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:23.703819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00bec2a-aa9b-412e-9f36-4660778c66f2 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:23.705296 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 492/980] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:23.705944 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 489/981] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:23 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:23.710750 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af174cd2-50ae-4013-819c-03bbb0ce90ea None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:23.711029 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af174cd2-50ae-4013-819c-03bbb0ce90ea None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:23.711307 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af174cd2-50ae-4013-819c-03bbb0ce90ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:23.715645 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f81318-1316-42eb-8819-072e95523272 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:23.731087 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-af174cd2-50ae-4013-819c-03bbb0ce90ea None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:23.732312 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-af174cd2-50ae-4013-819c-03bbb0ce90ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:23.819695 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 493/982] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:23 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:23.864512 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f81318-1316-42eb-8819-072e95523272 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:23.923455 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2ef80b-914c-4c9d-8d9c-4b6dc510a50c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:23.924564 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2ef80b-914c-4c9d-8d9c-4b6dc510a50c None admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:23.925060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2ef80b-914c-4c9d-8d9c-4b6dc510a50c None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:23.925550 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2ef80b-914c-4c9d-8d9c-4b6dc510a50c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:23.927213 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2ef80b-914c-4c9d-8d9c-4b6dc510a50c None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:23.929768 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2ef80b-914c-4c9d-8d9c-4b6dc510a50c None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:23.972150 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 490/983] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:44:23 2022] POST /identity/v3/endpoints => generated 349 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:44:24.005024 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af174cd2-50ae-4013-819c-03bbb0ce90ea None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:24.099149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2ef80b-914c-4c9d-8d9c-4b6dc510a50c None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:24.120305 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:24.137648 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:24.180775 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af174cd2-50ae-4013-819c-03bbb0ce90ea None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:24.245334 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36c0284-91a3-4964-981a-7ec51c5171c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:24.246281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36c0284-91a3-4964-981a-7ec51c5171c2 None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:24.246856 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36c0284-91a3-4964-981a-7ec51c5171c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:24.247349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36c0284-91a3-4964-981a-7ec51c5171c2 None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:24.248840 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36c0284-91a3-4964-981a-7ec51c5171c2 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:24.249180 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2ef80b-914c-4c9d-8d9c-4b6dc510a50c None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:24.250697 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36c0284-91a3-4964-981a-7ec51c5171c2 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:24.274526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 494/984] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 18 00:44:24 2022] POST /identity/v3/services => generated 242 bytes in 268 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:44:24.298221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad19187-cbb0-43fc-af5b-4e9a941df8a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:24.298940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad19187-cbb0-43fc-af5b-4e9a941df8a4 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:24.299221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad19187-cbb0-43fc-af5b-4e9a941df8a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:24.299792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad19187-cbb0-43fc-af5b-4e9a941df8a4 None admin] PATH_INFO: `/v3/services/image` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:24.300113 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad19187-cbb0-43fc-af5b-4e9a941df8a4 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:24.301843 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad19187-cbb0-43fc-af5b-4e9a941df8a4 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:24.312796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-9ad19187-cbb0-43fc-af5b-4e9a941df8a4 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 18 00:44:24.312796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:24.312796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:44:24.312796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:44:24.312796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:44:24.312796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:44:24.312796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:44:24.312796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:24.312796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 18 00:44:24.312796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:24.312796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:44:24.312796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:24.312796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:24.312796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:24.312796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:24.312796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:24.312796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:24.313876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:24.313876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:24.313876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:24.313876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:24.313876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:24.313876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:24.313876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:44:24.313876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:44:24.313876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:44:24.313876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:44:24.313876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:24.313876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:24.313876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:24.313876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:24.313876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:24.313876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:24.314850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:24.314850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:24.314850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:24.314850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:24.314850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:24.314850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:24.314850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:24.314850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:24.314850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:24.314850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:24.314850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:44:24.314850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:24.314850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 18 00:44:24.314850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:24.318549 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 491/985] 10.222.0.42 () {56 vars in 1105 bytes} [Sun Sep 18 00:44:24 2022] GET /identity/v3/services/image => generated 86 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:24.320223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36c0284-91a3-4964-981a-7ec51c5171c2 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:24.446812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36c0284-91a3-4964-981a-7ec51c5171c2 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:24.469717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:24.494320 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0e24f8-00d4-4e5e-84bb-6ad7ebef2a74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:24.495554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0e24f8-00d4-4e5e-84bb-6ad7ebef2a74 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:24.495780 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0e24f8-00d4-4e5e-84bb-6ad7ebef2a74 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:24.496056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0e24f8-00d4-4e5e-84bb-6ad7ebef2a74 None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:24.496705 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0e24f8-00d4-4e5e-84bb-6ad7ebef2a74 None admin] RBAC: Adding query filter params (name=image) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:24.497076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0e24f8-00d4-4e5e-84bb-6ad7ebef2a74 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:24.498494 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0e24f8-00d4-4e5e-84bb-6ad7ebef2a74 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:24.506108 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 495/986] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:24 2022] GET /identity/v3/services?name=image => generated 122 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:24.513643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad19187-cbb0-43fc-af5b-4e9a941df8a4 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:24.611213 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:24.632495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad19187-cbb0-43fc-af5b-4e9a941df8a4 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:24.667655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefac138-1f4d-4d98-8b93-df638f811f0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:24.668355 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefac138-1f4d-4d98-8b93-df638f811f0c None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:24.668698 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefac138-1f4d-4d98-8b93-df638f811f0c None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:24.669030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefac138-1f4d-4d98-8b93-df638f811f0c None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:24.669733 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefac138-1f4d-4d98-8b93-df638f811f0c None admin] RBAC: Adding query filter params (type=image) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:24.670099 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefac138-1f4d-4d98-8b93-df638f811f0c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:24.671567 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefac138-1f4d-4d98-8b93-df638f811f0c None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:24.676949 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 492/987] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:24 2022] GET /identity/v3/services?type=image => generated 122 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:26.714679 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a64582-1eab-4f1b-a2a8-347edc2a8e46 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:26.715298 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a64582-1eab-4f1b-a2a8-347edc2a8e46 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:26.715298 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a64582-1eab-4f1b-a2a8-347edc2a8e46 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:26.716827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 496/988] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:44:26.722230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c4a9e8-8229-4be2-ac38-6510b56b2017 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:26.722689 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c4a9e8-8229-4be2-ac38-6510b56b2017 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:26.722973 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c4a9e8-8229-4be2-ac38-6510b56b2017 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:26.745681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-d0c4a9e8-8229-4be2-ac38-6510b56b2017 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:26.747119 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d0c4a9e8-8229-4be2-ac38-6510b56b2017 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:26.776764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3258a98c-d678-4f49-a5ae-224f8f73ef49 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:26.777008 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3258a98c-d678-4f49-a5ae-224f8f73ef49 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:26.777202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3258a98c-d678-4f49-a5ae-224f8f73ef49 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:26.778596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 497/989] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:44:26.784114 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cba3ba5-48e4-4f07-8b7e-4502ed6d9015 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:26.784618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cba3ba5-48e4-4f07-8b7e-4502ed6d9015 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:26.784771 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cba3ba5-48e4-4f07-8b7e-4502ed6d9015 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:26.804110 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-8cba3ba5-48e4-4f07-8b7e-4502ed6d9015 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:26.805371 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-8cba3ba5-48e4-4f07-8b7e-4502ed6d9015 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:26.846123 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 493/990] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:26 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:26.898860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 498/991] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:26 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 18 00:44:26.975963 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcd71f5-010e-43aa-876c-0edcfec8c0d9 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:26.976395 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcd71f5-010e-43aa-876c-0edcfec8c0d9 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:26.976553 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcd71f5-010e-43aa-876c-0edcfec8c0d9 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:26.978139 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 494/992] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:26.984106 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae85527e-636f-41e2-bb0d-42746bad69ab None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:26.984477 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae85527e-636f-41e2-bb0d-42746bad69ab None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:26.984647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae85527e-636f-41e2-bb0d-42746bad69ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:27.004173 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-ae85527e-636f-41e2-bb0d-42746bad69ab None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:27.005377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-ae85527e-636f-41e2-bb0d-42746bad69ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:27.080959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 499/993] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:26 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 18 00:44:27.120059 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcd71f5-010e-43aa-876c-0edcfec8c0d9 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:27.211071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:27.241461 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcd71f5-010e-43aa-876c-0edcfec8c0d9 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:27.284644 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d02ca5-b472-4abc-8949-7edecb577afd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:27.285411 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d02ca5-b472-4abc-8949-7edecb577afd None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:27.285692 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d02ca5-b472-4abc-8949-7edecb577afd None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:27.285967 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d02ca5-b472-4abc-8949-7edecb577afd None admin] PATH_INFO: `/v3/services/compute` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:27.286659 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d02ca5-b472-4abc-8949-7edecb577afd None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:27.288230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d02ca5-b472-4abc-8949-7edecb577afd None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:27.299083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-80d02ca5-b472-4abc-8949-7edecb577afd None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 18 00:44:27.299083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:27.299083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:44:27.299083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:44:27.299083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:44:27.299083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:44:27.299083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:44:27.299083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:27.299083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 18 00:44:27.299083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:27.299083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:44:27.299083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:27.299083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:27.299083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:27.299083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:27.299083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:27.299083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:27.301015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:27.301015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:27.301015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:27.301015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:27.301015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:27.301015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:27.301015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:44:27.301015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:44:27.301015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:44:27.301015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:44:27.301015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:27.301015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:27.301015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:27.301015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:27.301015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:27.301015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:27.303719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:27.303719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:27.303719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:27.303719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:27.303719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:27.303719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:27.303719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:27.303719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:27.303719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:27.303719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:27.303719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:44:27.303719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:27.303719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 18 00:44:27.303719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:27.305158 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae85527e-636f-41e2-bb0d-42746bad69ab None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:27.306018 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 495/994] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:44:27 2022] GET /identity/v3/services/compute => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:27.307117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d02ca5-b472-4abc-8949-7edecb577afd None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:27.364245 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:27.431365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d02ca5-b472-4abc-8949-7edecb577afd None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:27.437900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae85527e-636f-41e2-bb0d-42746bad69ab None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:27.473368 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced77082-9004-40d4-80ae-8c3bb9bf7483 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:27.474329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87905fe3-08ee-4a34-b467-45af9fab76be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:27.474633 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced77082-9004-40d4-80ae-8c3bb9bf7483 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:27.474988 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87905fe3-08ee-4a34-b467-45af9fab76be None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:27.475104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced77082-9004-40d4-80ae-8c3bb9bf7483 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:27.475268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87905fe3-08ee-4a34-b467-45af9fab76be None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:27.475540 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced77082-9004-40d4-80ae-8c3bb9bf7483 None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:27.475628 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87905fe3-08ee-4a34-b467-45af9fab76be None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:27.476140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87905fe3-08ee-4a34-b467-45af9fab76be None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:27.476676 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced77082-9004-40d4-80ae-8c3bb9bf7483 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:27.477224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced77082-9004-40d4-80ae-8c3bb9bf7483 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:27.477539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87905fe3-08ee-4a34-b467-45af9fab76be None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:27.480344 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced77082-9004-40d4-80ae-8c3bb9bf7483 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:27.487544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-87905fe3-08ee-4a34-b467-45af9fab76be None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 18 00:44:27.487544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:27.487544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:44:27.487544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:44:27.487544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:44:27.487544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:44:27.487544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:44:27.487544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:27.487544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 18 00:44:27.487544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:27.487544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:44:27.487544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:27.487544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:27.487544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:27.487544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:27.487544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:27.487544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:27.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:27.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:27.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:27.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:27.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:27.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:27.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:44:27.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:44:27.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:44:27.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:44:27.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:27.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:27.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:27.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:27.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:27.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:27.489833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:27.489833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:27.489833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:27.489833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:27.489833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:27.489833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:27.489833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:27.489833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:27.489833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:27.489833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:27.489833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:44:27.489833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:27.489833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 18 00:44:27.489833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:27.489833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 500/995] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Sep 18 00:44:27 2022] GET /identity/v3/services/object-store => generated 93 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:27.490681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 496/996] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 18 00:44:27 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:27.490754 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87905fe3-08ee-4a34-b467-45af9fab76be None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:27.495387 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced77082-9004-40d4-80ae-8c3bb9bf7483 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:27.499846 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:27.607566 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87905fe3-08ee-4a34-b467-45af9fab76be None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:27.613444 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced77082-9004-40d4-80ae-8c3bb9bf7483 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:27.615315 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:27.643194 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf7ff29-b087-4aa6-8a1b-32a8f21a0677 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:27.643815 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf7ff29-b087-4aa6-8a1b-32a8f21a0677 None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:27.644083 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf7ff29-b087-4aa6-8a1b-32a8f21a0677 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:27.644358 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf7ff29-b087-4aa6-8a1b-32a8f21a0677 None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:27.645015 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf7ff29-b087-4aa6-8a1b-32a8f21a0677 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:27.646386 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf7ff29-b087-4aa6-8a1b-32a8f21a0677 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:27.648598 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78a6fc5-44a9-4f2a-8556-db444ee391e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:27.649212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78a6fc5-44a9-4f2a-8556-db444ee391e8 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:27.649494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78a6fc5-44a9-4f2a-8556-db444ee391e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:27.649768 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78a6fc5-44a9-4f2a-8556-db444ee391e8 None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:27.650403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78a6fc5-44a9-4f2a-8556-db444ee391e8 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:27.651278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78a6fc5-44a9-4f2a-8556-db444ee391e8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:27.652821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78a6fc5-44a9-4f2a-8556-db444ee391e8 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:27.658767 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 497/997] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Sep 18 00:44:27 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:27.659661 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78a6fc5-44a9-4f2a-8556-db444ee391e8 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:27.662748 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 501/998] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 18 00:44:27 2022] POST /identity/v3/services => generated 242 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:44:27.665425 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf7ff29-b087-4aa6-8a1b-32a8f21a0677 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:27.754401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:27.802133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78a6fc5-44a9-4f2a-8556-db444ee391e8 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:27.811250 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf7ff29-b087-4aa6-8a1b-32a8f21a0677 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:27.856928 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af3f84c-d226-4e3a-bbc9-38cbab691b97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:27.858325 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af3f84c-d226-4e3a-bbc9-38cbab691b97 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:27.858993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af3f84c-d226-4e3a-bbc9-38cbab691b97 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:27.859696 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af3f84c-d226-4e3a-bbc9-38cbab691b97 None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:27.860842 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af3f84c-d226-4e3a-bbc9-38cbab691b97 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:27.861557 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af3f84c-d226-4e3a-bbc9-38cbab691b97 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:27.864170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af3f84c-d226-4e3a-bbc9-38cbab691b97 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:27.873190 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 498/999] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 18 00:44:27 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:27.879434 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af3f84c-d226-4e3a-bbc9-38cbab691b97 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:27.881636 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fb7f7f-2003-46a6-a8ac-e1f8827f730a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:27.882335 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fb7f7f-2003-46a6-a8ac-e1f8827f730a None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:27.882652 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fb7f7f-2003-46a6-a8ac-e1f8827f730a None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:27.882961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fb7f7f-2003-46a6-a8ac-e1f8827f730a None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:27.884660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fb7f7f-2003-46a6-a8ac-e1f8827f730a None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:27.884965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fb7f7f-2003-46a6-a8ac-e1f8827f730a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:27.886970 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fb7f7f-2003-46a6-a8ac-e1f8827f730a None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:27.894334 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 502/1000] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Sep 18 00:44:27 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:27.903996 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fb7f7f-2003-46a6-a8ac-e1f8827f730a None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:27.938424 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:28.034834 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af3f84c-d226-4e3a-bbc9-38cbab691b97 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:28.047089 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fb7f7f-2003-46a6-a8ac-e1f8827f730a None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:28.067053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:28.074377 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5851ba6-1ff8-4401-9e37-8cd0ab0626f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:28.075029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5851ba6-1ff8-4401-9e37-8cd0ab0626f4 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:28.075361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5851ba6-1ff8-4401-9e37-8cd0ab0626f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:28.075748 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5851ba6-1ff8-4401-9e37-8cd0ab0626f4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:28.076466 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5851ba6-1ff8-4401-9e37-8cd0ab0626f4 None admin] RBAC: Adding query filter params (interface=public, service_id=6024c8b6ea3c454ebfcfcf7409fbe653, region_id=RegionOne) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:28.076811 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5851ba6-1ff8-4401-9e37-8cd0ab0626f4 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:28.078329 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5851ba6-1ff8-4401-9e37-8cd0ab0626f4 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:28.084442 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 499/1001] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Sep 18 00:44:27 2022] GET /identity/v3/endpoints?service_id=6024c8b6ea3c454ebfcfcf7409fbe653&interface=public®ion_id=RegionOne => generated 194 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:28.090189 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5851ba6-1ff8-4401-9e37-8cd0ab0626f4 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:28.107818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2281835-e94a-4f65-a9c3-f76f5422e51e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:28.108432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2281835-e94a-4f65-a9c3-f76f5422e51e None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:28.108740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2281835-e94a-4f65-a9c3-f76f5422e51e None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:28.108984 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2281835-e94a-4f65-a9c3-f76f5422e51e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:28.109694 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2281835-e94a-4f65-a9c3-f76f5422e51e None admin] RBAC: Adding query filter params (interface=admin, service_id=8af681d8c9ba478dba73dac151cb8b0c, region_id=RegionOne) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:28.109998 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2281835-e94a-4f65-a9c3-f76f5422e51e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:28.111408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2281835-e94a-4f65-a9c3-f76f5422e51e None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:28.116934 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 503/1002] 10.222.0.42 () {56 vars in 1249 bytes} [Sun Sep 18 00:44:27 2022] GET /identity/v3/endpoints?service_id=8af681d8c9ba478dba73dac151cb8b0c&interface=admin®ion_id=RegionOne => generated 193 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:28.122842 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2281835-e94a-4f65-a9c3-f76f5422e51e None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:28.227729 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5851ba6-1ff8-4401-9e37-8cd0ab0626f4 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:28.260622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:28.263719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4e02b1-1920-47d5-8d67-82db2425c8df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:28.264298 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4e02b1-1920-47d5-8d67-82db2425c8df None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:28.264575 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4e02b1-1920-47d5-8d67-82db2425c8df None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:28.264844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4e02b1-1920-47d5-8d67-82db2425c8df None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:28.265540 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4e02b1-1920-47d5-8d67-82db2425c8df None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:28.266984 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4e02b1-1920-47d5-8d67-82db2425c8df None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:28.276589 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 500/1003] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Sep 18 00:44:28 2022] GET /identity/v3/services => generated 1957 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:44:28.276809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2281835-e94a-4f65-a9c3-f76f5422e51e None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:28.324971 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f420e08-f8a6-470b-af28-cca2aba6cf36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:28.325718 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f420e08-f8a6-470b-af28-cca2aba6cf36 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:28.325999 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f420e08-f8a6-470b-af28-cca2aba6cf36 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:28.326268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f420e08-f8a6-470b-af28-cca2aba6cf36 None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:28.327098 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f420e08-f8a6-470b-af28-cca2aba6cf36 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:28.328959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f420e08-f8a6-470b-af28-cca2aba6cf36 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:28.338835 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 504/1004] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Sep 18 00:44:28 2022] GET /identity/v3/services => generated 1957 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:44:30.462104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a87ebf-cf93-4e8c-897e-e85f967529bb None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:30.462927 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a87ebf-cf93-4e8c-897e-e85f967529bb None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:30.462927 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a87ebf-cf93-4e8c-897e-e85f967529bb None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:30.465645 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 501/1005] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:30 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:44:30.475083 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74207dec-ade6-4662-af32-77ffa8ec2dca None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:30.475676 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74207dec-ade6-4662-af32-77ffa8ec2dca None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:30.476081 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74207dec-ade6-4662-af32-77ffa8ec2dca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:30.500730 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-74207dec-ade6-4662-af32-77ffa8ec2dca None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:30.502146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-74207dec-ade6-4662-af32-77ffa8ec2dca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:30.569689 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb595e2-a55c-4c30-9441-9721468e81d9 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:30.569955 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb595e2-a55c-4c30-9441-9721468e81d9 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:30.570228 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb595e2-a55c-4c30-9441-9721468e81d9 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:30.571910 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 502/1006] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:44:30.577401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e245823c-9d0d-4536-9fab-e6797b2d3404 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:30.577671 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e245823c-9d0d-4536-9fab-e6797b2d3404 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:30.577969 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e245823c-9d0d-4536-9fab-e6797b2d3404 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:30.589017 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 505/1007] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:30 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:30.592110 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000b3e00-5d32-4919-835c-b531c7ec48b9 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:30.592465 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000b3e00-5d32-4919-835c-b531c7ec48b9 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:30.592747 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000b3e00-5d32-4919-835c-b531c7ec48b9 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:30.594081 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 506/1008] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:44:30.599575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c469d57-42df-45cd-8958-cd55a0da198e None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:30.599923 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c469d57-42df-45cd-8958-cd55a0da198e None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:30.600169 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c469d57-42df-45cd-8958-cd55a0da198e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:30.601522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-e245823c-9d0d-4536-9fab-e6797b2d3404 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:30.603000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-e245823c-9d0d-4536-9fab-e6797b2d3404 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:30.622226 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-9c469d57-42df-45cd-8958-cd55a0da198e None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:30.623467 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-9c469d57-42df-45cd-8958-cd55a0da198e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:30.668232 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:30.688332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 503/1009] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:30 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:30.740943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 507/1010] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:30 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 18 00:44:30.868565 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e245823c-9d0d-4536-9fab-e6797b2d3404 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:31.005556 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c469d57-42df-45cd-8958-cd55a0da198e None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:31.023952 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e245823c-9d0d-4536-9fab-e6797b2d3404 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:31.043471 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:31.071389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299286e8-7ef4-41d8-9865-b25543e31073 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:31.072054 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299286e8-7ef4-41d8-9865-b25543e31073 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:31.072332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299286e8-7ef4-41d8-9865-b25543e31073 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:31.072650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299286e8-7ef4-41d8-9865-b25543e31073 None admin] PATH_INFO: `/v3/services/image` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:31.073269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299286e8-7ef4-41d8-9865-b25543e31073 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:31.074715 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299286e8-7ef4-41d8-9865-b25543e31073 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:31.084494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-299286e8-7ef4-41d8-9865-b25543e31073 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 18 00:44:31.084494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:31.084494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:44:31.084494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:44:31.084494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:44:31.084494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:44:31.084494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:44:31.084494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:31.084494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 18 00:44:31.084494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:31.084494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:44:31.084494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:31.084494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:31.084494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:31.084494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:31.084494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:31.084494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:44:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:44:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:44:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:44:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:31.086419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:31.086419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:31.086419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:31.086419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:31.086419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:31.086419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:31.086419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:31.086419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:31.086419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:31.086419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:31.086419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:44:31.086419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:31.086419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 18 00:44:31.086419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:31.086419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 504/1011] 10.222.0.42 () {56 vars in 1105 bytes} [Sun Sep 18 00:44:30 2022] GET /identity/v3/services/image => generated 86 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:31.087274 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299286e8-7ef4-41d8-9865-b25543e31073 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:31.157145 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:31.174393 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c469d57-42df-45cd-8958-cd55a0da198e None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:31.225743 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3c0af0-ca1c-49d0-a735-3f77befaab69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:31.226394 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3c0af0-ca1c-49d0-a735-3f77befaab69 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:31.226712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3c0af0-ca1c-49d0-a735-3f77befaab69 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:31.226988 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3c0af0-ca1c-49d0-a735-3f77befaab69 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:31.227594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3c0af0-ca1c-49d0-a735-3f77befaab69 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:31.229566 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3c0af0-ca1c-49d0-a735-3f77befaab69 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:31.236080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299286e8-7ef4-41d8-9865-b25543e31073 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:31.240277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-0a3c0af0-ca1c-49d0-a735-3f77befaab69 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 18 00:44:31.240277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:31.240277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:44:31.240277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:44:31.240277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:44:31.240277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:44:31.240277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:44:31.240277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:31.240277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 18 00:44:31.240277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:31.240277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:44:31.240277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:31.240277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:31.240277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:31.240277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:31.240277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:31.240277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:31.242219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:31.242219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:31.242219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:31.242219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:31.242219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:31.242219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:31.242219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:44:31.242219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:44:31.242219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:44:31.242219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:44:31.242219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:31.242219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:31.242219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:31.242219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:31.242219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:31.242219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:31.243836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:31.243836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:31.243836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:31.243836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:31.243836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:31.243836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:31.243836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:31.243836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:31.243836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:31.243836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:31.243836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:44:31.243836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:31.243836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 18 00:44:31.243836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:31.243836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 508/1012] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:44:31 2022] GET /identity/v3/services/compute => generated 88 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:31.245281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3c0af0-ca1c-49d0-a735-3f77befaab69 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:31.281941 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13daa77f-567e-410a-af1a-d344be14c42e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:31.282945 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13daa77f-567e-410a-af1a-d344be14c42e None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:31.283415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13daa77f-567e-410a-af1a-d344be14c42e None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:31.283885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13daa77f-567e-410a-af1a-d344be14c42e None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:31.285002 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13daa77f-567e-410a-af1a-d344be14c42e None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:31.287637 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13daa77f-567e-410a-af1a-d344be14c42e None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:31.305028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-13daa77f-567e-410a-af1a-d344be14c42e None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 18 00:44:31.305028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:31.305028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:44:31.305028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:44:31.305028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:44:31.305028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:44:31.305028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:44:31.305028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:31.305028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 18 00:44:31.305028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:31.305028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:44:31.305028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:31.305028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:31.305028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:31.305028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:31.305028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:31.305028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:31.306203 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:31.306203 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:31.306203 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:31.306203 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:31.306203 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:31.306203 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:31.306203 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:44:31.306203 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:44:31.306203 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:44:31.306203 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:44:31.306203 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:31.306203 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:31.306203 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:31.306203 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:31.306203 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:31.306203 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:31.307614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:31.307614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:31.307614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:31.307614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:31.307614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:31.307614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:31.307614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:31.307614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:31.307614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:31.307614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:31.307614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:44:31.307614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:31.307614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 18 00:44:31.307614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:31.307614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 505/1013] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Sep 18 00:44:31 2022] GET /identity/v3/services/object-store => generated 93 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:31.309019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13daa77f-567e-410a-af1a-d344be14c42e None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:31.337909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:31.362979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3c0af0-ca1c-49d0-a735-3f77befaab69 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:31.408156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c954c9c-d363-48c9-8bbd-7459675e0077 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:31.409187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c954c9c-d363-48c9-8bbd-7459675e0077 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:31.409666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c954c9c-d363-48c9-8bbd-7459675e0077 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:31.410150 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c954c9c-d363-48c9-8bbd-7459675e0077 None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:31.411215 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c954c9c-d363-48c9-8bbd-7459675e0077 None admin] RBAC: Adding query filter params (name=image) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:31.411730 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c954c9c-d363-48c9-8bbd-7459675e0077 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:31.414367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c954c9c-d363-48c9-8bbd-7459675e0077 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:31.422390 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 509/1014] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:31 2022] GET /identity/v3/services?name=image => generated 122 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:31.423276 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c954c9c-d363-48c9-8bbd-7459675e0077 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:31.436309 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13daa77f-567e-410a-af1a-d344be14c42e None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:31.456722 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:31.477768 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32be508c-90dd-4e8d-804e-a89ce8f52c4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:31.478433 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32be508c-90dd-4e8d-804e-a89ce8f52c4a None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:31.478685 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32be508c-90dd-4e8d-804e-a89ce8f52c4a None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:31.479546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32be508c-90dd-4e8d-804e-a89ce8f52c4a None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:31.480197 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32be508c-90dd-4e8d-804e-a89ce8f52c4a None admin] RBAC: Adding query filter params (name=compute) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:31.480500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32be508c-90dd-4e8d-804e-a89ce8f52c4a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:31.482100 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32be508c-90dd-4e8d-804e-a89ce8f52c4a None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:31.490009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 506/1015] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 18 00:44:31 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:31.490840 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32be508c-90dd-4e8d-804e-a89ce8f52c4a None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:31.553161 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c954c9c-d363-48c9-8bbd-7459675e0077 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:31.591424 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ecdec5-d1a8-4fdd-91b0-b513418baa7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:31.592546 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ecdec5-d1a8-4fdd-91b0-b513418baa7b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:31.593036 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ecdec5-d1a8-4fdd-91b0-b513418baa7b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:31.593517 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ecdec5-d1a8-4fdd-91b0-b513418baa7b None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:31.594686 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ecdec5-d1a8-4fdd-91b0-b513418baa7b None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:31.595165 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ecdec5-d1a8-4fdd-91b0-b513418baa7b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:31.597503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ecdec5-d1a8-4fdd-91b0-b513418baa7b None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:31.602835 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 510/1016] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Sep 18 00:44:31 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:31.604330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ecdec5-d1a8-4fdd-91b0-b513418baa7b None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:31.612358 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32be508c-90dd-4e8d-804e-a89ce8f52c4a None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:31.614001 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:31.669964 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29467ce3-317f-4bbf-a79d-a3358cecb495 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:31.670613 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29467ce3-317f-4bbf-a79d-a3358cecb495 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:31.670886 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29467ce3-317f-4bbf-a79d-a3358cecb495 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:31.671154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29467ce3-317f-4bbf-a79d-a3358cecb495 None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:31.671786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29467ce3-317f-4bbf-a79d-a3358cecb495 None admin] RBAC: Adding query filter params (type=image) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:31.672097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29467ce3-317f-4bbf-a79d-a3358cecb495 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:31.673454 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29467ce3-317f-4bbf-a79d-a3358cecb495 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:31.678897 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 507/1017] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:31 2022] GET /identity/v3/services?type=image => generated 350 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:31.679711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29467ce3-317f-4bbf-a79d-a3358cecb495 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:31.724849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ecdec5-d1a8-4fdd-91b0-b513418baa7b None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:31.742945 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:31.759543 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8647a7e0-6a31-4757-8af4-888bcdbd03be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:31.760689 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8647a7e0-6a31-4757-8af4-888bcdbd03be None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:31.760966 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8647a7e0-6a31-4757-8af4-888bcdbd03be None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:31.761234 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8647a7e0-6a31-4757-8af4-888bcdbd03be None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:31.761860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8647a7e0-6a31-4757-8af4-888bcdbd03be None admin] RBAC: Adding query filter params (type=compute) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:31.762171 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8647a7e0-6a31-4757-8af4-888bcdbd03be None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:31.763598 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8647a7e0-6a31-4757-8af4-888bcdbd03be None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:31.768792 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 511/1018] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 18 00:44:31 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:31.769623 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8647a7e0-6a31-4757-8af4-888bcdbd03be None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:31.814291 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29467ce3-317f-4bbf-a79d-a3358cecb495 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:31.851751 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b982d2cd-928a-4a40-8721-754aadc55089 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:31.852379 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b982d2cd-928a-4a40-8721-754aadc55089 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:31.852656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b982d2cd-928a-4a40-8721-754aadc55089 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:31.852927 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b982d2cd-928a-4a40-8721-754aadc55089 None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:31.853577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b982d2cd-928a-4a40-8721-754aadc55089 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:31.853925 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b982d2cd-928a-4a40-8721-754aadc55089 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:31.855378 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b982d2cd-928a-4a40-8721-754aadc55089 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:31.860879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 508/1019] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Sep 18 00:44:31 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:31.861725 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b982d2cd-928a-4a40-8721-754aadc55089 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:31.862027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:31.917746 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8647a7e0-6a31-4757-8af4-888bcdbd03be None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:31.957045 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dfa4c7e-1a08-4ef4-8989-cae316008aa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:31.957666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfa4c7e-1a08-4ef4-8989-cae316008aa0 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:31.957944 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfa4c7e-1a08-4ef4-8989-cae316008aa0 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:31.958216 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfa4c7e-1a08-4ef4-8989-cae316008aa0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:31.958966 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dfa4c7e-1a08-4ef4-8989-cae316008aa0 None admin] RBAC: Adding query filter params (interface=public, service_id=46d5b3a70c834f48961800549b1c5eb8, region_id=RegionOne) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:31.959301 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dfa4c7e-1a08-4ef4-8989-cae316008aa0 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:31.960707 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dfa4c7e-1a08-4ef4-8989-cae316008aa0 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:31.967006 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 512/1020] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Sep 18 00:44:31 2022] GET /identity/v3/endpoints?service_id=46d5b3a70c834f48961800549b1c5eb8&interface=public®ion_id=RegionOne => generated 194 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:31.967868 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dfa4c7e-1a08-4ef4-8989-cae316008aa0 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:31.998754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b982d2cd-928a-4a40-8721-754aadc55089 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:32.008346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:32.038523 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5fc997b-48fe-418e-81ba-827a58cfa0b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:32.039146 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fc997b-48fe-418e-81ba-827a58cfa0b7 None admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:32.039428 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fc997b-48fe-418e-81ba-827a58cfa0b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:32.039703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fc997b-48fe-418e-81ba-827a58cfa0b7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:32.040381 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5fc997b-48fe-418e-81ba-827a58cfa0b7 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:32.041784 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5fc997b-48fe-418e-81ba-827a58cfa0b7 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:32.064570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 509/1021] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:44:31 2022] POST /identity/v3/endpoints => generated 334 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:44:32.065149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5fc997b-48fe-418e-81ba-827a58cfa0b7 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:32.097710 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dfa4c7e-1a08-4ef4-8989-cae316008aa0 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:32.144927 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:32.153397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24c7beb-7ac6-4124-bbb8-11af3660f853 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:32.154427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24c7beb-7ac6-4124-bbb8-11af3660f853 None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:32.154742 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24c7beb-7ac6-4124-bbb8-11af3660f853 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:32.155019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24c7beb-7ac6-4124-bbb8-11af3660f853 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:32.155763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24c7beb-7ac6-4124-bbb8-11af3660f853 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:32.157269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24c7beb-7ac6-4124-bbb8-11af3660f853 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:32.182513 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 513/1022] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:44:31 2022] POST /identity/v3/endpoints => generated 325 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:44:32.187653 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5fc997b-48fe-418e-81ba-827a58cfa0b7 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:32.239324 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65839670-2bbc-41fe-9f95-f7333d3b5945 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:32.239977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65839670-2bbc-41fe-9f95-f7333d3b5945 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:32.240260 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65839670-2bbc-41fe-9f95-f7333d3b5945 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:32.240531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65839670-2bbc-41fe-9f95-f7333d3b5945 None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:32.241190 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65839670-2bbc-41fe-9f95-f7333d3b5945 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:32.242863 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65839670-2bbc-41fe-9f95-f7333d3b5945 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:32.251499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 510/1023] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Sep 18 00:44:32 2022] GET /identity/v3/services => generated 1957 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:44:34.397265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bc1750-c48d-4301-bd9f-02417f8d2c87 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:34.398104 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bc1750-c48d-4301-bd9f-02417f8d2c87 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:34.398104 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bc1750-c48d-4301-bd9f-02417f8d2c87 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:34.402513 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 514/1024] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:34.404940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b05e6ea-7833-4bb9-96d4-26e7b900b878 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:34.405473 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b05e6ea-7833-4bb9-96d4-26e7b900b878 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:34.405910 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b05e6ea-7833-4bb9-96d4-26e7b900b878 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:34.407792 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384f1822-814e-4f1e-acbd-3120fee03f1f None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:34.408128 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384f1822-814e-4f1e-acbd-3120fee03f1f None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:34.408296 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 511/1025] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:34 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:34.408438 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384f1822-814e-4f1e-acbd-3120fee03f1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:34.414255 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6711c59-6f17-4879-b75a-3f0af86e3308 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:34.415022 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6711c59-6f17-4879-b75a-3f0af86e3308 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:34.415182 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6711c59-6f17-4879-b75a-3f0af86e3308 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:34.428755 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-384f1822-814e-4f1e-acbd-3120fee03f1f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:34.429957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-384f1822-814e-4f1e-acbd-3120fee03f1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:34.437082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-d6711c59-6f17-4879-b75a-3f0af86e3308 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:34.438738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d6711c59-6f17-4879-b75a-3f0af86e3308 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:34.459250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:34.503402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 515/1026] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:34 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 18 00:44:34.504687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0985de-f719-47b5-931e-fe6742d1d298 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:34.505043 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0985de-f719-47b5-931e-fe6742d1d298 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:34.505306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0985de-f719-47b5-931e-fe6742d1d298 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:34.512736 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 516/1027] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:34 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:34.512736 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37edee1e-ed39-4691-b8d1-3418a317dc0b None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:34.512736 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37edee1e-ed39-4691-b8d1-3418a317dc0b None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:34.513286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37edee1e-ed39-4691-b8d1-3418a317dc0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:34.529385 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 512/1028] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:34 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 18 00:44:34.531854 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-37edee1e-ed39-4691-b8d1-3418a317dc0b None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:34.533146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-37edee1e-ed39-4691-b8d1-3418a317dc0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:34.607377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 517/1029] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:34 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 18 00:44:34.783370 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6711c59-6f17-4879-b75a-3f0af86e3308 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:34.882179 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37edee1e-ed39-4691-b8d1-3418a317dc0b None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:34.914099 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6711c59-6f17-4879-b75a-3f0af86e3308 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:34.951549 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:34.954673 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b3da0a-2fa9-4961-a1a8-e36e2d2c9ba6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:34.955720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b3da0a-2fa9-4961-a1a8-e36e2d2c9ba6 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:34.956000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b3da0a-2fa9-4961-a1a8-e36e2d2c9ba6 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:34.956272 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b3da0a-2fa9-4961-a1a8-e36e2d2c9ba6 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:34.957453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b3da0a-2fa9-4961-a1a8-e36e2d2c9ba6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:34.958916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b3da0a-2fa9-4961-a1a8-e36e2d2c9ba6 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:34.960622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 513/1030] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:34 2022] GET /identity/v3/domains/default => generated 199 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:34.961498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b3da0a-2fa9-4961-a1a8-e36e2d2c9ba6 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:34.987808 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:35.011351 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37edee1e-ed39-4691-b8d1-3418a317dc0b None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:35.048917 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aff59c5-1311-466b-b6ee-00d6d08ffd16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:35.049471 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aff59c5-1311-466b-b6ee-00d6d08ffd16 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:35.049756 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aff59c5-1311-466b-b6ee-00d6d08ffd16 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:35.050014 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aff59c5-1311-466b-b6ee-00d6d08ffd16 None admin] PATH_INFO: `/v3/services/image` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:35.050629 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aff59c5-1311-466b-b6ee-00d6d08ffd16 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:35.052259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aff59c5-1311-466b-b6ee-00d6d08ffd16 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:35.062711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-7aff59c5-1311-466b-b6ee-00d6d08ffd16 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 18 00:44:35.062711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:35.062711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:44:35.062711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:44:35.062711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:44:35.062711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:44:35.062711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:44:35.062711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:35.062711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 18 00:44:35.062711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:35.062711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:44:35.062711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:35.062711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:35.062711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:35.062711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:35.062711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:35.062711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:35.069914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:35.069914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:35.069914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:35.069914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:35.069914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:35.069914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:35.069914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:44:35.069914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:44:35.069914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:44:35.069914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:44:35.069914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:35.069914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:35.069914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:35.069914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:35.069914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:35.069914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:35.071040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:35.071040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:35.071040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:35.071040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:35.071040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:35.071040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:35.071040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:35.071040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:35.071040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:35.071040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:35.071040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:44:35.071040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:44:35.071040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 18 00:44:35.071040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:35.071040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 518/1031] 10.222.0.42 () {56 vars in 1105 bytes} [Sun Sep 18 00:44:34 2022] GET /identity/v3/services/image => generated 86 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:35.072031 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aff59c5-1311-466b-b6ee-00d6d08ffd16 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:35.088849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b3da0a-2fa9-4961-a1a8-e36e2d2c9ba6 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:35.141468 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5a857b-dda1-446b-95b0-787427cac744 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:35.142109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5a857b-dda1-446b-95b0-787427cac744 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:35.142413 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5a857b-dda1-446b-95b0-787427cac744 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:35.142754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5a857b-dda1-446b-95b0-787427cac744 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:35.154333 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5a857b-dda1-446b-95b0-787427cac744 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:44:35.159277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-be5a857b-dda1-446b-95b0-787427cac744 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 18 00:44:35.159277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:35.159277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:35.159277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:35.159277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:35.159277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:35.159277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:35.159277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:35.159277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:35.159277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:35.159277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:35.159277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:35.159277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:44:35.159277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:44:35.159277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:44:35.159277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:44:35.160591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:44:35.160591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:44:35.160591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:44:35.160591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:44:35.160591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:35.160591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:35.160591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:35.160591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:35.160591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:35.160591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:35.160591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:35.160591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:35.160591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:35.160591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:35.160591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:35.160591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:35.162016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:35.162016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:35.162016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:35.162016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:35.162016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:44:35.162016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:44:35.162016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:44:35.162016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:44:35.162016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:44:35.162016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:44:35.162016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 18 00:44:35.162016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:35.162016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 514/1032] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Sep 18 00:44:34 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:35.162016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5a857b-dda1-446b-95b0-787427cac744 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:35.179448 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aff59c5-1311-466b-b6ee-00d6d08ffd16 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:35.194369 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:35.218123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f5ef5e-a93a-4c6b-bb08-75ce60ea6877 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:35.218790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f5ef5e-a93a-4c6b-bb08-75ce60ea6877 None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:35.219075 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f5ef5e-a93a-4c6b-bb08-75ce60ea6877 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:35.219347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f5ef5e-a93a-4c6b-bb08-75ce60ea6877 None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:35.220013 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f5ef5e-a93a-4c6b-bb08-75ce60ea6877 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:35.221469 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f5ef5e-a93a-4c6b-bb08-75ce60ea6877 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:35.259213 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 519/1033] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 18 00:44:35 2022] POST /identity/v3/projects => generated 311 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:44:35.261278 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f5ef5e-a93a-4c6b-bb08-75ce60ea6877 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:35.280008 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:35.306251 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5a857b-dda1-446b-95b0-787427cac744 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:35.355765 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e0264c-0739-4310-80d1-4b35abca3437 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:35.356430 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e0264c-0739-4310-80d1-4b35abca3437 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:35.356713 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e0264c-0739-4310-80d1-4b35abca3437 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:35.356986 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e0264c-0739-4310-80d1-4b35abca3437 None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:35.357649 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e0264c-0739-4310-80d1-4b35abca3437 None admin] RBAC: Adding query filter params (name=image) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:35.357953 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e0264c-0739-4310-80d1-4b35abca3437 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:35.359502 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e0264c-0739-4310-80d1-4b35abca3437 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:35.371184 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 515/1034] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:35 2022] GET /identity/v3/services?name=image => generated 122 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:35.372868 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e0264c-0739-4310-80d1-4b35abca3437 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:35.433474 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f5ef5e-a93a-4c6b-bb08-75ce60ea6877 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:35.480204 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede73ae6-a673-435c-b958-c3117cddb2d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:35.481317 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede73ae6-a673-435c-b958-c3117cddb2d6 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:35.481909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede73ae6-a673-435c-b958-c3117cddb2d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:35.482426 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede73ae6-a673-435c-b958-c3117cddb2d6 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:35.483802 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede73ae6-a673-435c-b958-c3117cddb2d6 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:35.484439 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede73ae6-a673-435c-b958-c3117cddb2d6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:35.487055 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede73ae6-a673-435c-b958-c3117cddb2d6 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:35.498834 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 520/1035] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Sep 18 00:44:35 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:35.505112 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede73ae6-a673-435c-b958-c3117cddb2d6 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:35.533509 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e0264c-0739-4310-80d1-4b35abca3437 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:35.543983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:35.590109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cee77f-c820-40e6-a693-e0ab2f3b7182 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:35.590661 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cee77f-c820-40e6-a693-e0ab2f3b7182 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:35.591018 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cee77f-c820-40e6-a693-e0ab2f3b7182 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:35.591339 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cee77f-c820-40e6-a693-e0ab2f3b7182 None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:35.591992 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cee77f-c820-40e6-a693-e0ab2f3b7182 None admin] RBAC: Adding query filter params (type=image) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:35.592382 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cee77f-c820-40e6-a693-e0ab2f3b7182 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:35.593797 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cee77f-c820-40e6-a693-e0ab2f3b7182 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:35.599654 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 516/1036] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:35 2022] GET /identity/v3/services?type=image => generated 350 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:35.605745 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cee77f-c820-40e6-a693-e0ab2f3b7182 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:35.620945 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:35.650011 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede73ae6-a673-435c-b958-c3117cddb2d6 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:35.690104 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33356de3-9c1c-4c0a-9fe2-df543a5f459c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:35.690921 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33356de3-9c1c-4c0a-9fe2-df543a5f459c None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:35.691159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33356de3-9c1c-4c0a-9fe2-df543a5f459c None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:35.691528 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33356de3-9c1c-4c0a-9fe2-df543a5f459c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:35.704491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33356de3-9c1c-4c0a-9fe2-df543a5f459c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:35.705971 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33356de3-9c1c-4c0a-9fe2-df543a5f459c None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:35.730003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-33356de3-9c1c-4c0a-9fe2-df543a5f459c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:35.730003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:35.730003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:35.730003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:35.730003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:35.730003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:35.730003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:35.730003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:35.730003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:35.730003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:35.730003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:35.730003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:35.730003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:44:35.730003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:44:35.730003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:44:35.730003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:44:35.731107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:35.731107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:35.731107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:35.731107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:35.731107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:35.731107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:35.731107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:35.731107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:35.731107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:35.731107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:35.731107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:35.731107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:35.731107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:35.731107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:35.731107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:35.731107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:35.731107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:44:35.732025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:44:35.732025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:35.732025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:35.732206 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 521/1037] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:35.741308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33356de3-9c1c-4c0a-9fe2-df543a5f459c None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:35.764980 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cee77f-c820-40e6-a693-e0ab2f3b7182 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:35.823677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831314c4-e926-497b-be2c-a143f34a992c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:35.824410 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831314c4-e926-497b-be2c-a143f34a992c None admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:35.824708 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831314c4-e926-497b-be2c-a143f34a992c None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:35.825136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831314c4-e926-497b-be2c-a143f34a992c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:35.825832 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831314c4-e926-497b-be2c-a143f34a992c None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:35.827699 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831314c4-e926-497b-be2c-a143f34a992c None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:35.850509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:35.855876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 517/1038] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:44:35 2022] POST /identity/v3/endpoints => generated 327 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:44:35.884580 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33356de3-9c1c-4c0a-9fe2-df543a5f459c None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:35.946785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308152c4-5d1a-4040-a5db-2eb2367c3ebf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:35.947416 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308152c4-5d1a-4040-a5db-2eb2367c3ebf None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:35.948232 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308152c4-5d1a-4040-a5db-2eb2367c3ebf None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:35.948508 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308152c4-5d1a-4040-a5db-2eb2367c3ebf None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:35.949176 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308152c4-5d1a-4040-a5db-2eb2367c3ebf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:35.949483 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308152c4-5d1a-4040-a5db-2eb2367c3ebf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:35.950947 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308152c4-5d1a-4040-a5db-2eb2367c3ebf None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:35.966284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 522/1039] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:44:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:35.972335 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831314c4-e926-497b-be2c-a143f34a992c None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:36.067211 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:36.132609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831314c4-e926-497b-be2c-a143f34a992c None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:36.186665 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573f1852-09a7-4d46-8cdc-d92f6b645ece None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:36.187337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573f1852-09a7-4d46-8cdc-d92f6b645ece None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:36.187622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573f1852-09a7-4d46-8cdc-d92f6b645ece None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:36.187896 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573f1852-09a7-4d46-8cdc-d92f6b645ece None admin] PATH_INFO: `/v3/users/nova` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:36.198014 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573f1852-09a7-4d46-8cdc-d92f6b645ece None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:36.199546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573f1852-09a7-4d46-8cdc-d92f6b645ece None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:36.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-573f1852-09a7-4d46-8cdc-d92f6b645ece None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 18 00:44:36.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:36.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:36.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:36.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:36.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:36.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:36.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:36.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:36.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:36.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:36.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:36.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:44:36.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:44:36.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:44:36.216835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:44:36.217908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:36.217908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:36.217908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:44:36.217908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:44:36.217908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:44:36.217908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:44:36.217908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:36.217908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:36.217908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:36.217908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:36.217908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:36.217908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:36.217908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:36.217908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:36.217908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:36.217908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:36.217908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:36.217908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:36.218949 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:36.218949 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:36.218949 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:44:36.218949 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:44:36.218949 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:44:36.218949 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:44:36.218949 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:44:36.218949 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:44:36.218949 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 18 00:44:36.218949 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:36.218949 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 518/1040] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Sep 18 00:44:35 2022] GET /identity/v3/users/nova => generated 82 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:36.225725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308152c4-5d1a-4040-a5db-2eb2367c3ebf None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:36.352032 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308152c4-5d1a-4040-a5db-2eb2367c3ebf None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:36.390427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0a5dd8-11f5-4d16-adfb-ecbf532e24f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:36.390967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0a5dd8-11f5-4d16-adfb-ecbf532e24f6 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:36.391248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0a5dd8-11f5-4d16-adfb-ecbf532e24f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:36.391651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0a5dd8-11f5-4d16-adfb-ecbf532e24f6 None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:36.392310 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0a5dd8-11f5-4d16-adfb-ecbf532e24f6 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:36.392568 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0a5dd8-11f5-4d16-adfb-ecbf532e24f6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:36.394022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0a5dd8-11f5-4d16-adfb-ecbf532e24f6 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:36.397153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:36.425310 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 523/1041] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Sep 18 00:44:36 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:36.427655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573f1852-09a7-4d46-8cdc-d92f6b645ece None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:36.581612 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573f1852-09a7-4d46-8cdc-d92f6b645ece None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:36.616983 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:36.628985 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53403d10-43e3-4e67-b118-8110db7ce65f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:36.629618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53403d10-43e3-4e67-b118-8110db7ce65f None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:36.629899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53403d10-43e3-4e67-b118-8110db7ce65f None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:36.630169 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53403d10-43e3-4e67-b118-8110db7ce65f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:36.710780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53403d10-43e3-4e67-b118-8110db7ce65f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:36.712170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53403d10-43e3-4e67-b118-8110db7ce65f None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:36.823754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 519/1042] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:44:36 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 18 00:44:36.830266 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0a5dd8-11f5-4d16-adfb-ecbf532e24f6 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:36.984622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0a5dd8-11f5-4d16-adfb-ecbf532e24f6 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:37.032412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2d15c9-086e-46e3-bce4-58ddfe73cf72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:37.033098 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2d15c9-086e-46e3-bce4-58ddfe73cf72 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:37.033373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2d15c9-086e-46e3-bce4-58ddfe73cf72 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:37.033639 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2d15c9-086e-46e3-bce4-58ddfe73cf72 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:37.047456 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2d15c9-086e-46e3-bce4-58ddfe73cf72 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:37.049391 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2d15c9-086e-46e3-bce4-58ddfe73cf72 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:37.067863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-df2d15c9-086e-46e3-bce4-58ddfe73cf72 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:37.067863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:37.067863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:37.067863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:37.067863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:37.067863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:37.067863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:37.067863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:37.067863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:37.067863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:37.067863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:37.067863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:37.067863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:44:37.067863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:44:37.067863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:44:37.067863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:44:37.069699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:37.069699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:37.069699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:37.069699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:37.069699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:37.069699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:37.069699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:37.069699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:37.069699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:37.069699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:37.069699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:37.069699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:37.069699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:37.069699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:37.069699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:37.069699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:37.069699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:44:37.072333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:44:37.072333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:37.072333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:37.074586 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 524/1043] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:37.078158 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53403d10-43e3-4e67-b118-8110db7ce65f None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:37.123478 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:37.254003 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53403d10-43e3-4e67-b118-8110db7ce65f None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:37.314350 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68ca140-3c02-4b45-881c-7ba32cd1b977 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:37.315367 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68ca140-3c02-4b45-881c-7ba32cd1b977 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:37.315832 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68ca140-3c02-4b45-881c-7ba32cd1b977 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:37.316308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68ca140-3c02-4b45-881c-7ba32cd1b977 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:37.317466 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68ca140-3c02-4b45-881c-7ba32cd1b977 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:37.318024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68ca140-3c02-4b45-881c-7ba32cd1b977 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:37.320675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68ca140-3c02-4b45-881c-7ba32cd1b977 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:37.342764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 520/1044] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:44:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:37.351761 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2d15c9-086e-46e3-bce4-58ddfe73cf72 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:37.433332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:37.532315 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2d15c9-086e-46e3-bce4-58ddfe73cf72 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:37.596751 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab11b782-c2f3-4e4f-8705-8d9c3fee5f5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:37.597591 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab11b782-c2f3-4e4f-8705-8d9c3fee5f5d None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:37.597904 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab11b782-c2f3-4e4f-8705-8d9c3fee5f5d None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:37.598181 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab11b782-c2f3-4e4f-8705-8d9c3fee5f5d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:37.607941 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab11b782-c2f3-4e4f-8705-8d9c3fee5f5d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:37.609878 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab11b782-c2f3-4e4f-8705-8d9c3fee5f5d None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:37.625596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-ab11b782-c2f3-4e4f-8705-8d9c3fee5f5d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:44:37.625596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:37.625596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:37.625596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:37.625596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:37.625596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:37.625596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:37.625596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:37.625596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:37.625596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:37.625596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:37.625596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:37.625596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:44:37.625596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:44:37.625596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:44:37.635855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:44:37.635855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:37.635855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:37.635855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:37.635855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:37.635855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:37.635855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:37.635855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:37.635855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:37.635855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:37.635855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:37.635855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:37.635855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:37.635855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:37.635855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:37.635855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:37.635855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:37.637491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:44:37.637491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:44:37.637491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:44:37.637491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:44:37.637491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:44:37.637491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:44:37.637491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:44:37.637491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:37.637491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 525/1045] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:44:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:37.640467 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68ca140-3c02-4b45-881c-7ba32cd1b977 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:37.665586 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4bee66-3e9d-454f-969a-209265cf7a97 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:37.666004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4bee66-3e9d-454f-969a-209265cf7a97 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:37.666743 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4bee66-3e9d-454f-969a-209265cf7a97 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:37.668466 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 526/1046] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:44:37.673904 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88811d75-c11d-45c6-ae31-031861c64d3e None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:37.674192 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88811d75-c11d-45c6-ae31-031861c64d3e None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:37.674502 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88811d75-c11d-45c6-ae31-031861c64d3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:37.696228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-88811d75-c11d-45c6-ae31-031861c64d3e None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:37.698523 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-88811d75-c11d-45c6-ae31-031861c64d3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:37.777512 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:37.797693 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 527/1047] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:37 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 18 00:44:37.806956 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68ca140-3c02-4b45-881c-7ba32cd1b977 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:37.855558 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9aac58e-f27b-4512-8675-b4a331ecc9f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:37.856720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9aac58e-f27b-4512-8675-b4a331ecc9f2 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:37.857212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9aac58e-f27b-4512-8675-b4a331ecc9f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:37.857690 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9aac58e-f27b-4512-8675-b4a331ecc9f2 None admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:37.864465 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9aac58e-f27b-4512-8675-b4a331ecc9f2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:37.864465 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9aac58e-f27b-4512-8675-b4a331ecc9f2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:37.864465 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9aac58e-f27b-4512-8675-b4a331ecc9f2 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:37.879668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 521/1048] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:44:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:37.885738 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88811d75-c11d-45c6-ae31-031861c64d3e None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:37.937460 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:38.038826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88811d75-c11d-45c6-ae31-031861c64d3e None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:38.100643 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463d5a9e-e534-48a3-b093-b9b9ef243bf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:38.101690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463d5a9e-e534-48a3-b093-b9b9ef243bf2 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:38.102000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463d5a9e-e534-48a3-b093-b9b9ef243bf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:38.102516 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463d5a9e-e534-48a3-b093-b9b9ef243bf2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:38.104505 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463d5a9e-e534-48a3-b093-b9b9ef243bf2 None admin] RBAC: Adding query filter params (role.id=88c42ae856ff4e2d8a42a5b1d50b97f9, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=a59d73fd45a54651930d52300704fdb0) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:38.104944 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463d5a9e-e534-48a3-b093-b9b9ef243bf2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:38.107741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463d5a9e-e534-48a3-b093-b9b9ef243bf2 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:38.127088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 528/1049] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:44:37 2022] GET /identity/v3/role_assignments?user.id=a59d73fd45a54651930d52300704fdb0&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=88c42ae856ff4e2d8a42a5b1d50b97f9 => generated 259 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:38.128601 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9aac58e-f27b-4512-8675-b4a331ecc9f2 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:38.158009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c60054-b9c3-426b-bf32-bc84db238d62 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:38.158330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c60054-b9c3-426b-bf32-bc84db238d62 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:38.158729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c60054-b9c3-426b-bf32-bc84db238d62 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:38.160040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 529/1050] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:44:38.166289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606faeac-c0c4-4341-a238-f0d788eb0a73 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:38.166641 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606faeac-c0c4-4341-a238-f0d788eb0a73 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:38.166873 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606faeac-c0c4-4341-a238-f0d788eb0a73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:38.187409 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-606faeac-c0c4-4341-a238-f0d788eb0a73 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:38.188736 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-606faeac-c0c4-4341-a238-f0d788eb0a73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:38.281150 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9aac58e-f27b-4512-8675-b4a331ecc9f2 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:38.290586 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 530/1051] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 18 00:44:38.334491 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e1a7b3-1300-4a62-a5ad-c208d2cfdff7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:38.335412 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e1a7b3-1300-4a62-a5ad-c208d2cfdff7 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:38.335834 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e1a7b3-1300-4a62-a5ad-c208d2cfdff7 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:38.337384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e1a7b3-1300-4a62-a5ad-c208d2cfdff7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:38.338093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e1a7b3-1300-4a62-a5ad-c208d2cfdff7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:38.340716 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e1a7b3-1300-4a62-a5ad-c208d2cfdff7 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:38.343394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 522/1052] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:38.349917 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606faeac-c0c4-4341-a238-f0d788eb0a73 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:38.402687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:38.533432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606faeac-c0c4-4341-a238-f0d788eb0a73 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:38.573879 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ad6988-04f1-458e-846c-9818e068f64f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:38.574513 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ad6988-04f1-458e-846c-9818e068f64f None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:38.574822 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ad6988-04f1-458e-846c-9818e068f64f None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:38.575098 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ad6988-04f1-458e-846c-9818e068f64f None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:38.575788 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ad6988-04f1-458e-846c-9818e068f64f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:38.577271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ad6988-04f1-458e-846c-9818e068f64f None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:38.609939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 531/1053] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 18 00:44:38 2022] POST /identity/v3/users => generated 285 bytes in 255 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:44:38.638341 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e1a7b3-1300-4a62-a5ad-c208d2cfdff7 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:38.667828 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:38.763727 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e1a7b3-1300-4a62-a5ad-c208d2cfdff7 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:38.804623 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6618ed8d-0f39-43dc-ba39-2509bf515336 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:38.805311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6618ed8d-0f39-43dc-ba39-2509bf515336 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:38.805596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6618ed8d-0f39-43dc-ba39-2509bf515336 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:38.805872 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6618ed8d-0f39-43dc-ba39-2509bf515336 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:38.905237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6618ed8d-0f39-43dc-ba39-2509bf515336 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:38.907016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6618ed8d-0f39-43dc-ba39-2509bf515336 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:38.930362 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:39.082540 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 523/1054] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:44:38 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 441 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 18 00:44:39.086378 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ad6988-04f1-458e-846c-9818e068f64f None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:39.247151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:39.271451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ad6988-04f1-458e-846c-9818e068f64f None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:39.328253 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e7b666-29cd-4427-8994-d4142cc29b73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:39.329028 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e7b666-29cd-4427-8994-d4142cc29b73 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:39.329314 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e7b666-29cd-4427-8994-d4142cc29b73 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:39.329614 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e7b666-29cd-4427-8994-d4142cc29b73 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:39.350645 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e7b666-29cd-4427-8994-d4142cc29b73 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:39.353413 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e7b666-29cd-4427-8994-d4142cc29b73 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:39.378902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-90e7b666-29cd-4427-8994-d4142cc29b73 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:39.378902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:39.378902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:39.378902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:39.378902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:39.378902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:39.378902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:39.378902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:39.378902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:39.378902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:39.378902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:39.378902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:39.378902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:44:39.378902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:44:39.378902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:44:39.378902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:44:39.380347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:39.380347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:39.380347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:39.380347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:39.380347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:39.380347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:39.380347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:39.380347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:39.380347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:39.380347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:39.380347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:39.380347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:39.380347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:39.380347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:39.380347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:39.380347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:39.380347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:44:39.381257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:44:39.381257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:39.381257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:39.382218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 532/1055] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:39.390869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6618ed8d-0f39-43dc-ba39-2509bf515336 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:39.554277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6618ed8d-0f39-43dc-ba39-2509bf515336 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:39.575260 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:39.603717 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae3f184-3d0e-4161-9a54-0a1083e66208 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:39.604487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae3f184-3d0e-4161-9a54-0a1083e66208 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:39.604773 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae3f184-3d0e-4161-9a54-0a1083e66208 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:39.605184 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae3f184-3d0e-4161-9a54-0a1083e66208 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:39.606116 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae3f184-3d0e-4161-9a54-0a1083e66208 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:39.606424 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae3f184-3d0e-4161-9a54-0a1083e66208 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:39.608430 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae3f184-3d0e-4161-9a54-0a1083e66208 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:39.630778 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 524/1056] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:44:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:40.525065 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9380132-4fa5-4436-a598-a894b7b603d2 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:40.525065 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9380132-4fa5-4436-a598-a894b7b603d2 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:40.525065 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9380132-4fa5-4436-a598-a894b7b603d2 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:40.526261 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 533/1057] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:40.532023 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eb2aa9-b10a-474c-95e0-e746e2364ab2 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:40.532511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eb2aa9-b10a-474c-95e0-e746e2364ab2 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:40.532597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eb2aa9-b10a-474c-95e0-e746e2364ab2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:40.553084 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-74eb2aa9-b10a-474c-95e0-e746e2364ab2 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:40.554294 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-74eb2aa9-b10a-474c-95e0-e746e2364ab2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:40.624359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 525/1058] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:40.897031 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9380132-4fa5-4436-a598-a894b7b603d2 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:41.008267 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9380132-4fa5-4436-a598-a894b7b603d2 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:41.047142 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e05ec5-8b27-46e8-846c-4a4d59f5d5c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:41.047805 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e05ec5-8b27-46e8-846c-4a4d59f5d5c1 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:41.048083 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e05ec5-8b27-46e8-846c-4a4d59f5d5c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:41.048355 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e05ec5-8b27-46e8-846c-4a4d59f5d5c1 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:41.056957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:41.062740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e05ec5-8b27-46e8-846c-4a4d59f5d5c1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:44:41.067916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-55e05ec5-8b27-46e8-846c-4a4d59f5d5c1 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 18 00:44:41.067916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:41.067916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:41.067916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:41.067916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:41.067916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:41.067916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:41.067916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:41.067916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:41.067916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:41.067916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:41.067916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:41.067916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:44:41.067916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:44:41.067916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:44:41.067916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:44:41.069824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:44:41.069824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:44:41.069824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:44:41.069824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:44:41.069824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:41.069824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:41.069824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:41.069824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:41.069824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:41.069824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:41.069824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:41.069824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:41.069824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:41.069824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:41.069824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:41.069824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:41.070795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:41.070795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:41.070795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:41.070795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:41.070795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:44:41.070795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:44:41.070795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:44:41.070795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:44:41.070795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:44:41.070795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:44:41.070795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 18 00:44:41.070795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:41.070795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 534/1059] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Sep 18 00:44:40 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:41.075283 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eb2aa9-b10a-474c-95e0-e746e2364ab2 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:41.160009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23342b9d-c3d0-43d3-b212-a0b8fe55d324 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:41.160194 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23342b9d-c3d0-43d3-b212-a0b8fe55d324 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:41.160471 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23342b9d-c3d0-43d3-b212-a0b8fe55d324 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:41.161866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 535/1060] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:44:41.167974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687f3707-8285-4b4d-9ecf-f15b7ab08dbb None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:41.168252 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687f3707-8285-4b4d-9ecf-f15b7ab08dbb None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:41.168538 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687f3707-8285-4b4d-9ecf-f15b7ab08dbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:41.181149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eb2aa9-b10a-474c-95e0-e746e2364ab2 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:41.183574 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:41.187397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-687f3707-8285-4b4d-9ecf-f15b7ab08dbb None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:41.188613 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-687f3707-8285-4b4d-9ecf-f15b7ab08dbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:41.217440 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025b008e-ce81-4cfe-8f59-e22bba380899 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:41.218048 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025b008e-ce81-4cfe-8f59-e22bba380899 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:41.218321 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025b008e-ce81-4cfe-8f59-e22bba380899 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:41.218631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025b008e-ce81-4cfe-8f59-e22bba380899 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:41.219299 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-025b008e-ce81-4cfe-8f59-e22bba380899 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:41.219591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-025b008e-ce81-4cfe-8f59-e22bba380899 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:41.220949 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-025b008e-ce81-4cfe-8f59-e22bba380899 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:41.230522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 526/1061] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Sep 18 00:44:41 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:41.236418 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025b008e-ce81-4cfe-8f59-e22bba380899 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:41.261075 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 536/1062] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:41 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 18 00:44:41.350928 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025b008e-ce81-4cfe-8f59-e22bba380899 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:41.387775 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae46f81f-62ad-4adf-b7a0-53ae98f39902 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:41.388414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae46f81f-62ad-4adf-b7a0-53ae98f39902 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:41.388694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae46f81f-62ad-4adf-b7a0-53ae98f39902 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:41.388970 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae46f81f-62ad-4adf-b7a0-53ae98f39902 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:41.400074 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae46f81f-62ad-4adf-b7a0-53ae98f39902 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:41.401549 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae46f81f-62ad-4adf-b7a0-53ae98f39902 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:41.417595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-ae46f81f-62ad-4adf-b7a0-53ae98f39902 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:41.417595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:41.417595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:41.417595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:41.417595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:41.417595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:41.417595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:41.417595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:41.417595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:41.417595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:41.417595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:41.417595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:41.417595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:44:41.417595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:44:41.417595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:44:41.417595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:44:41.419610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:41.419610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:41.419610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:41.419610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:41.419610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:41.419610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:41.419610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:41.419610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:41.419610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:41.419610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:41.419610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:41.419610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:41.419610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:41.419610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:41.419610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:41.419610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:41.419610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:44:41.421242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:44:41.421242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:41.421242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:41.422296 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 527/1063] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:41.425035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687f3707-8285-4b4d-9ecf-f15b7ab08dbb None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:41.533498 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687f3707-8285-4b4d-9ecf-f15b7ab08dbb None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:41.540914 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae46f81f-62ad-4adf-b7a0-53ae98f39902 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:41.562628 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:41.562834 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:41.574287 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb69d49-144a-4d3f-a476-e131a95340c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:41.574933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb69d49-144a-4d3f-a476-e131a95340c7 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:41.575217 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb69d49-144a-4d3f-a476-e131a95340c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:41.575493 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb69d49-144a-4d3f-a476-e131a95340c7 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:41.576153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb69d49-144a-4d3f-a476-e131a95340c7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:41.576480 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb69d49-144a-4d3f-a476-e131a95340c7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:41.577877 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb69d49-144a-4d3f-a476-e131a95340c7 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:41.590752 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 537/1064] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:44:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:41.600053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb69d49-144a-4d3f-a476-e131a95340c7 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:41.676668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae46f81f-62ad-4adf-b7a0-53ae98f39902 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:41.706771 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb69d49-144a-4d3f-a476-e131a95340c7 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:41.711040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a73984c-48d1-429d-a0bf-d690b87dc35d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:41.711680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a73984c-48d1-429d-a0bf-d690b87dc35d None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:41.711959 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a73984c-48d1-429d-a0bf-d690b87dc35d None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:41.712229 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a73984c-48d1-429d-a0bf-d690b87dc35d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:41.723265 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a73984c-48d1-429d-a0bf-d690b87dc35d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:44:41.729374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-6a73984c-48d1-429d-a0bf-d690b87dc35d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:44:41.729374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:41.729374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:41.729374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:41.729374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:41.729374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:41.729374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:41.729374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:41.729374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:41.729374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:41.729374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:41.729374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:41.729374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:44:41.729374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:44:41.729374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:44:41.729374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:44:41.730500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:44:41.730500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:44:41.730500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:44:41.730500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:44:41.730500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:41.730500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:41.730500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:41.730500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:41.730500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:41.730500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:41.730500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:41.730500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:41.730500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:41.730500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:41.730500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:41.730500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:41.731731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:41.731731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:41.731731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:41.731731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:41.731731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:44:41.731731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:44:41.731731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:44:41.731731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:44:41.731731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:44:41.731731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:44:41.731731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:44:41.731731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:41.731731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 528/1065] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:44:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:41.737198 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a73984c-48d1-429d-a0bf-d690b87dc35d None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:41.745157 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa494e28-2f46-44bd-804b-e80c3bb69b03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:41.745791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa494e28-2f46-44bd-804b-e80c3bb69b03 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:41.746132 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa494e28-2f46-44bd-804b-e80c3bb69b03 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:41.746319 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa494e28-2f46-44bd-804b-e80c3bb69b03 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:41.754147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa494e28-2f46-44bd-804b-e80c3bb69b03 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:41.755635 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa494e28-2f46-44bd-804b-e80c3bb69b03 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:41.767198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-fa494e28-2f46-44bd-804b-e80c3bb69b03 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 18 00:44:41.767198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:41.767198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:41.767198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:41.767198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:41.767198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:41.767198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:41.767198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:41.767198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:41.767198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:41.767198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:41.767198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:41.767198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:44:41.767198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:44:41.767198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:44:41.767198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:44:41.768330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:41.768330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:41.768330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:44:41.768330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:44:41.768330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:44:41.768330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:44:41.768330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:41.768330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:41.768330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:41.768330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:41.768330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:41.768330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:41.768330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:41.768330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:41.768330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:41.768330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:41.768330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:41.768330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:41.769765 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:41.769765 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:41.769765 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:44:41.769765 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:44:41.769765 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:44:41.769765 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:44:41.769765 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:44:41.769765 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:44:41.769765 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 18 00:44:41.769765 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:41.769765 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 538/1066] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Sep 18 00:44:41 2022] GET /identity/v3/users/nova => generated 82 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:41.774035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa494e28-2f46-44bd-804b-e80c3bb69b03 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:41.823645 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:41.829202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:41.848849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a73984c-48d1-429d-a0bf-d690b87dc35d None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:41.882445 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa494e28-2f46-44bd-804b-e80c3bb69b03 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:41.885275 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc71252-01db-4606-9c3d-ea38ec037910 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:41.885893 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc71252-01db-4606-9c3d-ea38ec037910 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:41.886162 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc71252-01db-4606-9c3d-ea38ec037910 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:41.886428 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc71252-01db-4606-9c3d-ea38ec037910 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:41.887190 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc71252-01db-4606-9c3d-ea38ec037910 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:41.887488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc71252-01db-4606-9c3d-ea38ec037910 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:41.888875 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc71252-01db-4606-9c3d-ea38ec037910 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:41.899159 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 529/1067] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:44:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:41.905765 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc71252-01db-4606-9c3d-ea38ec037910 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:41.921046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6920e9f3-6e09-4c61-b0bb-8b845e49dcd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:41.921680 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6920e9f3-6e09-4c61-b0bb-8b845e49dcd7 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:41.921957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6920e9f3-6e09-4c61-b0bb-8b845e49dcd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:41.922256 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6920e9f3-6e09-4c61-b0bb-8b845e49dcd7 None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:41.923046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6920e9f3-6e09-4c61-b0bb-8b845e49dcd7 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:41.923350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6920e9f3-6e09-4c61-b0bb-8b845e49dcd7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:41.924807 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6920e9f3-6e09-4c61-b0bb-8b845e49dcd7 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:41.954278 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 539/1068] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Sep 18 00:44:41 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:41.955566 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6920e9f3-6e09-4c61-b0bb-8b845e49dcd7 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:42.013112 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc71252-01db-4606-9c3d-ea38ec037910 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:42.048737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a29c33-2598-4dd1-bea9-5b077b4c6698 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:42.049370 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a29c33-2598-4dd1-bea9-5b077b4c6698 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:42.049646 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a29c33-2598-4dd1-bea9-5b077b4c6698 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:42.049927 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a29c33-2598-4dd1-bea9-5b077b4c6698 None admin] PATH_INFO: `/v3/users/04461da56a8345a18af6ba0fa92cf089` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:42.059398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6920e9f3-6e09-4c61-b0bb-8b845e49dcd7 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:42.061710 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:42.065877 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a29c33-2598-4dd1-bea9-5b077b4c6698 None admin] RBAC: Authorizing `identity:get_user(user_id=04461da56a8345a18af6ba0fa92cf089)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:42.067373 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a29c33-2598-4dd1-bea9-5b077b4c6698 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:42.067547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:42.069365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 530/1069] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:44:41 2022] GET /identity/v3/users/04461da56a8345a18af6ba0fa92cf089 => generated 285 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:42.075387 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a29c33-2598-4dd1-bea9-5b077b4c6698 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:42.098354 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b382185-226b-4eda-ae0c-bc2588139b64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:42.099025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b382185-226b-4eda-ae0c-bc2588139b64 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:42.099302 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b382185-226b-4eda-ae0c-bc2588139b64 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:42.099562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b382185-226b-4eda-ae0c-bc2588139b64 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:42.110514 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b382185-226b-4eda-ae0c-bc2588139b64 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:42.111979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b382185-226b-4eda-ae0c-bc2588139b64 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:42.130268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-1b382185-226b-4eda-ae0c-bc2588139b64 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:42.130268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:42.130268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:42.130268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:42.130268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:42.130268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:42.130268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:42.130268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:42.130268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:42.130268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:42.130268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:42.130268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:42.130268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:44:42.130268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:44:42.130268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:44:42.130268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:44:42.131522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:42.131522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:42.131522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:42.131522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:42.131522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:42.131522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:42.131522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:42.131522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:42.131522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:42.131522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:42.131522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:42.131522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:42.131522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:42.131522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:42.131522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:42.131522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:42.131522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:44:42.132607 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:44:42.132607 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:42.132607 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:42.132607 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 540/1070] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:42.137123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b382185-226b-4eda-ae0c-bc2588139b64 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:42.191834 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a29c33-2598-4dd1-bea9-5b077b4c6698 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:42.230680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e9c699-0eb2-40fc-a19a-67a42d749137 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:42.231171 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e9c699-0eb2-40fc-a19a-67a42d749137 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:42.231445 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e9c699-0eb2-40fc-a19a-67a42d749137 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:42.231713 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e9c699-0eb2-40fc-a19a-67a42d749137 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:42.247413 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b382185-226b-4eda-ae0c-bc2588139b64 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:42.282688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340e12eb-7a0f-454b-bd0b-d8a3b53bc824 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:42.283353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340e12eb-7a0f-454b-bd0b-d8a3b53bc824 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:42.283618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340e12eb-7a0f-454b-bd0b-d8a3b53bc824 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:42.283910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340e12eb-7a0f-454b-bd0b-d8a3b53bc824 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:42.284554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340e12eb-7a0f-454b-bd0b-d8a3b53bc824 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:42.284853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340e12eb-7a0f-454b-bd0b-d8a3b53bc824 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:42.286224 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340e12eb-7a0f-454b-bd0b-d8a3b53bc824 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:42.298668 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 541/1071] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:44:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:42.301849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:42.304918 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340e12eb-7a0f-454b-bd0b-d8a3b53bc824 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:42.307700 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e9c699-0eb2-40fc-a19a-67a42d749137 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:42.309461 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e9c699-0eb2-40fc-a19a-67a42d749137 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:42.329558 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:42.421917 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340e12eb-7a0f-454b-bd0b-d8a3b53bc824 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:42.423275 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 531/1072] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:44:42 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 18 00:44:42.438920 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e9c699-0eb2-40fc-a19a-67a42d749137 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:42.469290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8fcea1b-9f41-43ac-8314-49b88a713c9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:42.469929 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fcea1b-9f41-43ac-8314-49b88a713c9e None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:42.470204 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fcea1b-9f41-43ac-8314-49b88a713c9e None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:42.470604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fcea1b-9f41-43ac-8314-49b88a713c9e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:42.483613 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8fcea1b-9f41-43ac-8314-49b88a713c9e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:42.487050 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8fcea1b-9f41-43ac-8314-49b88a713c9e None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:42.507284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-b8fcea1b-9f41-43ac-8314-49b88a713c9e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:44:42.507284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:42.507284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:42.507284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:42.507284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:42.507284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:42.507284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:42.507284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:42.507284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:42.507284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:42.507284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:42.507284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:42.507284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:44:42.507284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:44:42.507284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:44:42.509003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:44:42.509003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:42.509003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:42.509003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:42.509003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:42.509003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:42.509003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:42.509003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:42.509003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:42.509003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:42.509003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:42.509003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:42.509003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:42.509003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:42.509003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:42.509003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:42.509003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:42.510434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:44:42.510434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:44:42.510434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:44:42.510434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:44:42.510434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:44:42.510434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:44:42.510434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:44:42.510434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:42.510434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 542/1073] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:44:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:42.515846 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8fcea1b-9f41-43ac-8314-49b88a713c9e None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:42.564500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:42.569541 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e9c699-0eb2-40fc-a19a-67a42d749137 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:42.602541 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:42.609630 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d71cd2e-7784-4c98-9a8a-5ef357b3766c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:42.610271 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d71cd2e-7784-4c98-9a8a-5ef357b3766c None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:42.610578 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d71cd2e-7784-4c98-9a8a-5ef357b3766c None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:42.610863 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d71cd2e-7784-4c98-9a8a-5ef357b3766c None admin] PATH_INFO: `/v3/projects/afc8669902004a108a80b47f16395aae` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:42.612017 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d71cd2e-7784-4c98-9a8a-5ef357b3766c None admin] RBAC: Authorizing `identity:get_project(project_id=afc8669902004a108a80b47f16395aae)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:42.613466 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d71cd2e-7784-4c98-9a8a-5ef357b3766c None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:42.615192 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 532/1074] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:44:42 2022] GET /identity/v3/projects/afc8669902004a108a80b47f16395aae => generated 311 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:42.621521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d71cd2e-7784-4c98-9a8a-5ef357b3766c None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:42.627468 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8fcea1b-9f41-43ac-8314-49b88a713c9e None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:42.667385 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d878b21-0d29-4f29-8385-8a5e2bb4e944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:42.668002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d878b21-0d29-4f29-8385-8a5e2bb4e944 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:42.668282 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d878b21-0d29-4f29-8385-8a5e2bb4e944 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:42.668556 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d878b21-0d29-4f29-8385-8a5e2bb4e944 None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:42.669251 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d878b21-0d29-4f29-8385-8a5e2bb4e944 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:42.669562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d878b21-0d29-4f29-8385-8a5e2bb4e944 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:42.671037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d878b21-0d29-4f29-8385-8a5e2bb4e944 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:42.683896 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 543/1075] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:44:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:42.690189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d878b21-0d29-4f29-8385-8a5e2bb4e944 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:42.738564 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d71cd2e-7784-4c98-9a8a-5ef357b3766c None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:42.777413 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c77cb02-bb50-4b41-97c4-852fa377be0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:42.778025 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c77cb02-bb50-4b41-97c4-852fa377be0d None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:42.778304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c77cb02-bb50-4b41-97c4-852fa377be0d None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:42.778615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c77cb02-bb50-4b41-97c4-852fa377be0d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:42.779357 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c77cb02-bb50-4b41-97c4-852fa377be0d None admin] RBAC: Adding query filter params (role.id=1e04c9fea65244d28e1e59fc4c64f32f, scope.project.id=afc8669902004a108a80b47f16395aae, user.id=04461da56a8345a18af6ba0fa92cf089) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:42.779666 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c77cb02-bb50-4b41-97c4-852fa377be0d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:42.781124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c77cb02-bb50-4b41-97c4-852fa377be0d None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:42.786879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 533/1076] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:44:42 2022] GET /identity/v3/role_assignments?user.id=04461da56a8345a18af6ba0fa92cf089&scope.project.id=afc8669902004a108a80b47f16395aae&role.id=1e04c9fea65244d28e1e59fc4c64f32f => generated 259 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:42.827934 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d878b21-0d29-4f29-8385-8a5e2bb4e944 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:42.867147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20d23cd-7bbe-4a9d-bb4c-11378111d4af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:42.867947 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20d23cd-7bbe-4a9d-bb4c-11378111d4af None admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:42.868232 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20d23cd-7bbe-4a9d-bb4c-11378111d4af None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:42.868504 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20d23cd-7bbe-4a9d-bb4c-11378111d4af None admin] PATH_INFO: `/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/a59d73fd45a54651930d52300704fdb0/roles/88c42ae856ff4e2d8a42a5b1d50b97f9` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:42.871513 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20d23cd-7bbe-4a9d-bb4c-11378111d4af None admin] RBAC: Authorizing `identity:create_grant(project_id=c520a8da3dc54adba4f7cc04ef42e367, user_id=a59d73fd45a54651930d52300704fdb0, role_id=88c42ae856ff4e2d8a42a5b1d50b97f9)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:42.873023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20d23cd-7bbe-4a9d-bb4c-11378111d4af None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:42.884510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:42.891803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 544/1077] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:44:42 2022] PUT /identity/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/a59d73fd45a54651930d52300704fdb0/roles/88c42ae856ff4e2d8a42a5b1d50b97f9 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:44:44.969192 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343dd769-d095-409f-b834-8880c913979d None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:44.969792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343dd769-d095-409f-b834-8880c913979d None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:44.969792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343dd769-d095-409f-b834-8880c913979d None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:44.971150 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 534/1078] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:44.977180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d3cc60-957a-48e3-a1c5-ece869bfaaf5 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:44.977725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d3cc60-957a-48e3-a1c5-ece869bfaaf5 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:44.977725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d3cc60-957a-48e3-a1c5-ece869bfaaf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:44.997545 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-e4d3cc60-957a-48e3-a1c5-ece869bfaaf5 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:44.998831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-e4d3cc60-957a-48e3-a1c5-ece869bfaaf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:45.068111 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 545/1079] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:45.076702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e715132-f456-4065-962e-b53ff052290e None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:45.077072 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e715132-f456-4065-962e-b53ff052290e None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:45.077409 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e715132-f456-4065-962e-b53ff052290e None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:45.079034 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 535/1080] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:45.084965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd3f1f0-4bc5-4adb-8659-b0d828bd2ba1 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:45.085255 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd3f1f0-4bc5-4adb-8659-b0d828bd2ba1 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:45.085635 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd3f1f0-4bc5-4adb-8659-b0d828bd2ba1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:45.107571 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-fbd3f1f0-4bc5-4adb-8659-b0d828bd2ba1 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:45.109368 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-fbd3f1f0-4bc5-4adb-8659-b0d828bd2ba1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:45.179817 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 546/1081] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:45.338789 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e715132-f456-4065-962e-b53ff052290e None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:45.368827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:45.443019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e715132-f456-4065-962e-b53ff052290e None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:45.448919 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd3f1f0-4bc5-4adb-8659-b0d828bd2ba1 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:45.483205 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b2943f-e945-44c2-bf67-63a4169ae136 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:45.483967 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b2943f-e945-44c2-bf67-63a4169ae136 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:45.484330 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b2943f-e945-44c2-bf67-63a4169ae136 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:45.484686 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b2943f-e945-44c2-bf67-63a4169ae136 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:45.495828 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b2943f-e945-44c2-bf67-63a4169ae136 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:44:45.500656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-89b2943f-e945-44c2-bf67-63a4169ae136 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:44:45.500656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:45.500656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:45.500656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:45.500656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:45.500656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:45.500656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:45.500656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:45.500656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:45.500656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:45.500656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:45.500656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:45.500656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:44:45.500656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:44:45.500656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:44:45.500656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:44:45.501571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:44:45.501571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:44:45.501571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:44:45.501571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:44:45.501571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:45.501571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:45.501571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:45.501571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:45.501571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:45.501571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:45.501571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:45.501571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:45.501571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:45.501571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:45.501571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:45.501571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:45.502688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:45.502688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:45.502688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:45.502688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:45.502688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:44:45.502688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:44:45.502688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:44:45.502688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:44:45.502688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:44:45.502688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:44:45.502688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:44:45.502688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:45.502688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 536/1082] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:44:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:45.507588 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:45.507774 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b2943f-e945-44c2-bf67-63a4169ae136 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:45.551465 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd3f1f0-4bc5-4adb-8659-b0d828bd2ba1 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:45.591486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3426bfa-30bd-4b76-ba5b-8322ad7d7f56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:45.591978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3426bfa-30bd-4b76-ba5b-8322ad7d7f56 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:45.592255 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3426bfa-30bd-4b76-ba5b-8322ad7d7f56 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:45.592568 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3426bfa-30bd-4b76-ba5b-8322ad7d7f56 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:45.603094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3426bfa-30bd-4b76-ba5b-8322ad7d7f56 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:44:45.606734 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:45.608101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-b3426bfa-30bd-4b76-ba5b-8322ad7d7f56 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 18 00:44:45.608101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:45.608101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:45.608101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:45.608101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:45.608101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:45.608101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:45.608101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:45.608101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:45.608101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:45.608101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:45.608101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:45.608101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:44:45.608101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:44:45.608101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:44:45.608101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:44:45.609039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:44:45.609039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:44:45.609039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:44:45.609039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:44:45.609039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:45.609039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:45.609039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:45.609039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:45.609039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:45.609039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:45.609039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:45.609039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:45.609039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:45.609039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:45.609039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:45.609039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:45.609862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:45.609862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:45.609862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:45.609862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:45.609862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:44:45.609862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:44:45.609862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:44:45.609862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:44:45.609862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:44:45.609862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:44:45.609862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 18 00:44:45.609862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:45.609862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 547/1083] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Sep 18 00:44:45 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:45.616904 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3426bfa-30bd-4b76-ba5b-8322ad7d7f56 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:45.625715 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b2943f-e945-44c2-bf67-63a4169ae136 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:45.666183 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de351d8c-48db-44d6-8610-63da50f1c052 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:45.666812 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de351d8c-48db-44d6-8610-63da50f1c052 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:45.667084 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de351d8c-48db-44d6-8610-63da50f1c052 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:45.667349 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de351d8c-48db-44d6-8610-63da50f1c052 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:45.668016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de351d8c-48db-44d6-8610-63da50f1c052 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:45.668311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de351d8c-48db-44d6-8610-63da50f1c052 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:45.669680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de351d8c-48db-44d6-8610-63da50f1c052 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:45.678182 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 537/1084] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:44:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:45.684725 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de351d8c-48db-44d6-8610-63da50f1c052 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:45.726916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3426bfa-30bd-4b76-ba5b-8322ad7d7f56 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:45.746713 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:45.766648 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450b7656-c6ce-418d-ad7c-d4379509351b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:45.767250 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450b7656-c6ce-418d-ad7c-d4379509351b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:45.767527 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450b7656-c6ce-418d-ad7c-d4379509351b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:45.767800 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450b7656-c6ce-418d-ad7c-d4379509351b None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:45.768480 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450b7656-c6ce-418d-ad7c-d4379509351b None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:45.768784 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450b7656-c6ce-418d-ad7c-d4379509351b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:45.770174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450b7656-c6ce-418d-ad7c-d4379509351b None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:45.779613 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 548/1085] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Sep 18 00:44:45 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:45.785770 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450b7656-c6ce-418d-ad7c-d4379509351b None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:45.794651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de351d8c-48db-44d6-8610-63da50f1c052 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:45.832640 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d884d5f-5fe1-4a05-95fd-b9ea0a0b51fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:45.833276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d884d5f-5fe1-4a05-95fd-b9ea0a0b51fa None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:45.833561 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d884d5f-5fe1-4a05-95fd-b9ea0a0b51fa None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:45.833833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d884d5f-5fe1-4a05-95fd-b9ea0a0b51fa None admin] PATH_INFO: `/v3/users/04461da56a8345a18af6ba0fa92cf089` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:45.835014 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d884d5f-5fe1-4a05-95fd-b9ea0a0b51fa None admin] RBAC: Authorizing `identity:get_user(user_id=04461da56a8345a18af6ba0fa92cf089)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:45.836461 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d884d5f-5fe1-4a05-95fd-b9ea0a0b51fa None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:45.838256 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 538/1086] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:44:45 2022] GET /identity/v3/users/04461da56a8345a18af6ba0fa92cf089 => generated 285 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:45.846038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d884d5f-5fe1-4a05-95fd-b9ea0a0b51fa None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:45.871883 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:45.907503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450b7656-c6ce-418d-ad7c-d4379509351b None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:45.948018 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6926d4ca-e3dc-42fd-a982-5df39b6f118d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:45.948651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6926d4ca-e3dc-42fd-a982-5df39b6f118d None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:45.948926 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6926d4ca-e3dc-42fd-a982-5df39b6f118d None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:45.949192 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6926d4ca-e3dc-42fd-a982-5df39b6f118d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:45.960531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d884d5f-5fe1-4a05-95fd-b9ea0a0b51fa None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:45.960777 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6926d4ca-e3dc-42fd-a982-5df39b6f118d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:45.962134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6926d4ca-e3dc-42fd-a982-5df39b6f118d None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:45.977052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-6926d4ca-e3dc-42fd-a982-5df39b6f118d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:45.977052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:45.977052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:45.977052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:45.977052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:45.977052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:45.977052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:45.977052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:45.977052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:45.977052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:45.977052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:45.977052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:45.977052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:44:45.977052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:44:45.977052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:44:45.977052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:44:45.979269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:45.979269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:45.979269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:45.979269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:45.979269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:45.979269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:45.979269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:45.979269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:45.979269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:45.979269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:45.979269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:45.979269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:45.979269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:45.979269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:45.979269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:45.979269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:45.979269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:44:45.980403 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:44:45.980403 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:45.980403 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:45.980403 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 549/1087] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:45.984565 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6926d4ca-e3dc-42fd-a982-5df39b6f118d None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:45.999523 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96756e1b-344b-4152-b5ae-c3178e669a43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:46.000223 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96756e1b-344b-4152-b5ae-c3178e669a43 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:46.000446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96756e1b-344b-4152-b5ae-c3178e669a43 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:46.000718 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96756e1b-344b-4152-b5ae-c3178e669a43 None admin] PATH_INFO: `/v3/projects/afc8669902004a108a80b47f16395aae` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:46.001859 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96756e1b-344b-4152-b5ae-c3178e669a43 None admin] RBAC: Authorizing `identity:get_project(project_id=afc8669902004a108a80b47f16395aae)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:46.003364 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96756e1b-344b-4152-b5ae-c3178e669a43 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:46.005080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 539/1088] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:44:45 2022] GET /identity/v3/projects/afc8669902004a108a80b47f16395aae => generated 311 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:46.011178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96756e1b-344b-4152-b5ae-c3178e669a43 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:46.035040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:46.109060 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6926d4ca-e3dc-42fd-a982-5df39b6f118d None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:46.110872 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:46.124226 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96756e1b-344b-4152-b5ae-c3178e669a43 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:46.144670 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cbd7b0-177a-4732-99ee-69b1188b968d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:46.145276 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cbd7b0-177a-4732-99ee-69b1188b968d None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:46.145562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cbd7b0-177a-4732-99ee-69b1188b968d None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:46.145833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cbd7b0-177a-4732-99ee-69b1188b968d None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:46.146504 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1cbd7b0-177a-4732-99ee-69b1188b968d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:46.146810 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1cbd7b0-177a-4732-99ee-69b1188b968d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:46.148190 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1cbd7b0-177a-4732-99ee-69b1188b968d None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:46.160635 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 550/1089] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:44:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:46.164855 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd65e2d-39fc-4062-93f9-b0cdd0b3312b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:46.165563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd65e2d-39fc-4062-93f9-b0cdd0b3312b None admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:46.165801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd65e2d-39fc-4062-93f9-b0cdd0b3312b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:46.166077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd65e2d-39fc-4062-93f9-b0cdd0b3312b None admin] PATH_INFO: `/v3/projects/afc8669902004a108a80b47f16395aae/users/04461da56a8345a18af6ba0fa92cf089/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:46.167235 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cbd7b0-177a-4732-99ee-69b1188b968d None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:46.168614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd65e2d-39fc-4062-93f9-b0cdd0b3312b None admin] RBAC: Authorizing `identity:create_grant(project_id=afc8669902004a108a80b47f16395aae, user_id=04461da56a8345a18af6ba0fa92cf089, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:46.170049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd65e2d-39fc-4062-93f9-b0cdd0b3312b None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:46.185042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 540/1090] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:44:46 2022] PUT /identity/v3/projects/afc8669902004a108a80b47f16395aae/users/04461da56a8345a18af6ba0fa92cf089/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:44:46.270756 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:46.281547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cbd7b0-177a-4732-99ee-69b1188b968d None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:46.325844 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446a7846-b080-4a22-ad1f-a7fcb222b226 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:46.326543 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446a7846-b080-4a22-ad1f-a7fcb222b226 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:46.326826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446a7846-b080-4a22-ad1f-a7fcb222b226 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:46.327099 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446a7846-b080-4a22-ad1f-a7fcb222b226 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:46.335556 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446a7846-b080-4a22-ad1f-a7fcb222b226 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:46.337042 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446a7846-b080-4a22-ad1f-a7fcb222b226 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:46.350186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-446a7846-b080-4a22-ad1f-a7fcb222b226 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 18 00:44:46.350186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:46.350186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:46.350186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:46.350186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:46.350186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:46.350186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:46.350186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:46.350186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:46.350186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:46.350186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:46.350186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:46.350186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:44:46.350186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:44:46.350186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:44:46.350186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:44:46.351387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:46.351387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:46.351387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:44:46.351387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:44:46.351387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:44:46.351387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:44:46.351387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:46.351387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:46.351387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:46.351387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:46.351387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:46.351387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:46.351387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:46.351387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:46.351387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:46.351387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:46.351387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:46.351387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:46.352392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:46.352392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:46.352392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:44:46.352392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:44:46.352392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:44:46.352392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:44:46.352392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:44:46.352392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:44:46.352392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 18 00:44:46.352392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:46.352392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 551/1091] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Sep 18 00:44:46 2022] GET /identity/v3/users/nova => generated 82 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:46.358330 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd65e2d-39fc-4062-93f9-b0cdd0b3312b None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:46.478149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd65e2d-39fc-4062-93f9-b0cdd0b3312b None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:46.516214 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05cd33df-42bd-4aa3-b06f-c3f7fae66794 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:46.516839 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cd33df-42bd-4aa3-b06f-c3f7fae66794 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:46.517112 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cd33df-42bd-4aa3-b06f-c3f7fae66794 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:46.517387 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cd33df-42bd-4aa3-b06f-c3f7fae66794 None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:46.518132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05cd33df-42bd-4aa3-b06f-c3f7fae66794 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:46.518438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05cd33df-42bd-4aa3-b06f-c3f7fae66794 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:46.519938 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05cd33df-42bd-4aa3-b06f-c3f7fae66794 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:46.542034 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 541/1092] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Sep 18 00:44:46 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:46.548070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446a7846-b080-4a22-ad1f-a7fcb222b226 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:46.654446 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446a7846-b080-4a22-ad1f-a7fcb222b226 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:46.694283 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a66d7d-5679-4111-8fd4-89c814cdab18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:46.694920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a66d7d-5679-4111-8fd4-89c814cdab18 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:46.695193 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a66d7d-5679-4111-8fd4-89c814cdab18 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:46.695463 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a66d7d-5679-4111-8fd4-89c814cdab18 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:46.718863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:46.768302 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a66d7d-5679-4111-8fd4-89c814cdab18 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:46.769672 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a66d7d-5679-4111-8fd4-89c814cdab18 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:46.874091 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 552/1093] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:44:46 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 18 00:44:46.880708 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05cd33df-42bd-4aa3-b06f-c3f7fae66794 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:46.896052 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:47.007008 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05cd33df-42bd-4aa3-b06f-c3f7fae66794 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:47.042754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94316535-73a3-40a5-9f54-b8d99ce11ad9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:47.043304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94316535-73a3-40a5-9f54-b8d99ce11ad9 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:47.043570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94316535-73a3-40a5-9f54-b8d99ce11ad9 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:47.043838 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94316535-73a3-40a5-9f54-b8d99ce11ad9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:47.054443 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94316535-73a3-40a5-9f54-b8d99ce11ad9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:47.055911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94316535-73a3-40a5-9f54-b8d99ce11ad9 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:47.072243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-94316535-73a3-40a5-9f54-b8d99ce11ad9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:47.072243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:47.072243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:47.072243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:47.072243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:47.072243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:47.072243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:47.072243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:47.072243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:47.072243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:47.072243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:47.072243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:47.072243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:44:47.072243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:44:47.072243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:44:47.072243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:44:47.073279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:47.073279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:47.073279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:47.073279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:47.073279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:47.073279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:47.073279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:47.073279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:47.073279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:47.073279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:47.073279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:47.073279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:47.073279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:47.073279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:47.073279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:47.073279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:47.073279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:44:47.074172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:44:47.074172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:44:47.074172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:47.074172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 542/1094] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:44:47.079295 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a66d7d-5679-4111-8fd4-89c814cdab18 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:47.145739 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:47.220947 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a66d7d-5679-4111-8fd4-89c814cdab18 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:47.269940 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e60d45-25d8-4134-9725-e937b987ade0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:47.270596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e60d45-25d8-4134-9725-e937b987ade0 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:47.270983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e60d45-25d8-4134-9725-e937b987ade0 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:47.271267 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e60d45-25d8-4134-9725-e937b987ade0 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:47.271923 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e60d45-25d8-4134-9725-e937b987ade0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:47.272256 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e60d45-25d8-4134-9725-e937b987ade0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:47.273748 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e60d45-25d8-4134-9725-e937b987ade0 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:47.291667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 553/1095] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:44:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:47.297786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94316535-73a3-40a5-9f54-b8d99ce11ad9 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:47.368182 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:47.429969 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94316535-73a3-40a5-9f54-b8d99ce11ad9 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:47.464762 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9da536-b1a9-41b4-85a8-def5c2a8190b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:47.465397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9da536-b1a9-41b4-85a8-def5c2a8190b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:47.465670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9da536-b1a9-41b4-85a8-def5c2a8190b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:47.465939 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9da536-b1a9-41b4-85a8-def5c2a8190b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:47.472908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9da536-b1a9-41b4-85a8-def5c2a8190b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:47.474339 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9da536-b1a9-41b4-85a8-def5c2a8190b None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:47.487643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-9d9da536-b1a9-41b4-85a8-def5c2a8190b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:44:47.487643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:47.487643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:47.487643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:47.487643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:47.487643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:47.487643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:47.487643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:47.487643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:47.487643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:47.487643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:47.487643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:47.487643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:44:47.487643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:44:47.487643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:44:47.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:44:47.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:47.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:47.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:47.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:47.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:47.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:47.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:47.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:47.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:47.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:47.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:47.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:47.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:47.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:47.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:47.488609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:47.489516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:44:47.489516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:44:47.489516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:44:47.489516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:44:47.489516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:44:47.489516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:44:47.489516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:44:47.489516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:44:47.489516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 543/1096] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:44:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:47.494870 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e60d45-25d8-4134-9725-e937b987ade0 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:47.618817 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e60d45-25d8-4134-9725-e937b987ade0 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:47.642188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:47.669663 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa741cf-b5d6-4949-b485-a2418e7844a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:47.670274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa741cf-b5d6-4949-b485-a2418e7844a8 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:47.670585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa741cf-b5d6-4949-b485-a2418e7844a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:47.670866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa741cf-b5d6-4949-b485-a2418e7844a8 None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:47.671571 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa741cf-b5d6-4949-b485-a2418e7844a8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:47.671872 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa741cf-b5d6-4949-b485-a2418e7844a8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:47.673312 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa741cf-b5d6-4949-b485-a2418e7844a8 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:47.692762 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 554/1097] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:44:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:47.694109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9da536-b1a9-41b4-85a8-def5c2a8190b None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:47.810960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9da536-b1a9-41b4-85a8-def5c2a8190b None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:47.838034 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:47.856221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1066c073-bdd9-4418-987c-f69087feb230 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:47.856825 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1066c073-bdd9-4418-987c-f69087feb230 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:47.857101 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1066c073-bdd9-4418-987c-f69087feb230 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:47.857370 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1066c073-bdd9-4418-987c-f69087feb230 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:47.858101 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1066c073-bdd9-4418-987c-f69087feb230 None admin] RBAC: Adding query filter params (role.id=88c42ae856ff4e2d8a42a5b1d50b97f9, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=a59d73fd45a54651930d52300704fdb0) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:47.858408 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1066c073-bdd9-4418-987c-f69087feb230 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:47.859865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1066c073-bdd9-4418-987c-f69087feb230 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:47.865730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 544/1098] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:44:47 2022] GET /identity/v3/role_assignments?user.id=a59d73fd45a54651930d52300704fdb0&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=88c42ae856ff4e2d8a42a5b1d50b97f9 => generated 608 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:48.439521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c503c460-4462-4ae4-8e13-cd092f3eef5c None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:48.440286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c503c460-4462-4ae4-8e13-cd092f3eef5c None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:48.440422 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c503c460-4462-4ae4-8e13-cd092f3eef5c None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:48.441752 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 555/1099] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:48.447513 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e5578c-a20d-4736-a482-ebfade2a228f None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:48.447983 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e5578c-a20d-4736-a482-ebfade2a228f None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:48.448090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e5578c-a20d-4736-a482-ebfade2a228f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:48.470224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-03e5578c-a20d-4736-a482-ebfade2a228f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:48.471769 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-03e5578c-a20d-4736-a482-ebfade2a228f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:48.571523 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 545/1100] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:48.845487 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c503c460-4462-4ae4-8e13-cd092f3eef5c None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:48.976435 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c503c460-4462-4ae4-8e13-cd092f3eef5c None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:49.032436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b8837d-68b2-4f98-b3a3-8cc9cae05fba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:49.033067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b8837d-68b2-4f98-b3a3-8cc9cae05fba None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:49.033343 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b8837d-68b2-4f98-b3a3-8cc9cae05fba None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:49.033612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b8837d-68b2-4f98-b3a3-8cc9cae05fba None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:49.044667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b8837d-68b2-4f98-b3a3-8cc9cae05fba None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:44:49.049535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-66b8837d-68b2-4f98-b3a3-8cc9cae05fba None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:44:49.049535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:44:49.049535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:44:49.049535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:44:49.049535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:44:49.049535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:44:49.049535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:44:49.049535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:44:49.049535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:44:49.049535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:44:49.049535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:44:49.049535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:44:49.049535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:44:49.049535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:44:49.049535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:44:49.049535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:44:49.051052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:44:49.051052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:44:49.051052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:44:49.051052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:44:49.051052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:44:49.051052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:44:49.051052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:44:49.051052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:44:49.051052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:44:49.051052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:44:49.051052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:44:49.051052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:44:49.051052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:44:49.051052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:44:49.051052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:44:49.051052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:44:49.052609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:44:49.052609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:44:49.052609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:44:49.052609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:44:49.052609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:44:49.052609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:44:49.052609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:44:49.052609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:44:49.052609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:44:49.052609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:44:49.052609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:44:49.052609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:44:49.054623 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 556/1101] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:44:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:44:49.057771 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e5578c-a20d-4736-a482-ebfade2a228f None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:49.165342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e5578c-a20d-4736-a482-ebfade2a228f None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:49.182981 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:49.210955 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d2f3a0-1420-40f3-b5c6-6d22f2f369d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:49.211786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d2f3a0-1420-40f3-b5c6-6d22f2f369d6 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:49.212177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d2f3a0-1420-40f3-b5c6-6d22f2f369d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:49.212561 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d2f3a0-1420-40f3-b5c6-6d22f2f369d6 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:49.214016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d2f3a0-1420-40f3-b5c6-6d22f2f369d6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:49.214483 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d2f3a0-1420-40f3-b5c6-6d22f2f369d6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:49.216467 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d2f3a0-1420-40f3-b5c6-6d22f2f369d6 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:49.228142 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 546/1102] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:44:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:49.234790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b8837d-68b2-4f98-b3a3-8cc9cae05fba None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:49.284913 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:49.356483 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b8837d-68b2-4f98-b3a3-8cc9cae05fba None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:49.390952 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d700f7a-6dc2-49d1-867c-11dcd5a1485d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:49.391575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d700f7a-6dc2-49d1-867c-11dcd5a1485d None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:49.391859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d700f7a-6dc2-49d1-867c-11dcd5a1485d None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:49.392126 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d700f7a-6dc2-49d1-867c-11dcd5a1485d None admin] PATH_INFO: `/v3/users/04461da56a8345a18af6ba0fa92cf089` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:49.393229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d700f7a-6dc2-49d1-867c-11dcd5a1485d None admin] RBAC: Authorizing `identity:get_user(user_id=04461da56a8345a18af6ba0fa92cf089)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:49.394678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d700f7a-6dc2-49d1-867c-11dcd5a1485d None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:49.396342 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 557/1103] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:44:49 2022] GET /identity/v3/users/04461da56a8345a18af6ba0fa92cf089 => generated 285 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:44:49.405176 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d2f3a0-1420-40f3-b5c6-6d22f2f369d6 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:49.539950 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d2f3a0-1420-40f3-b5c6-6d22f2f369d6 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:49.585907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5696b578-cbb6-42bf-9e0d-dd8d4171d022 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:49.586873 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5696b578-cbb6-42bf-9e0d-dd8d4171d022 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:49.587314 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5696b578-cbb6-42bf-9e0d-dd8d4171d022 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:49.587754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5696b578-cbb6-42bf-9e0d-dd8d4171d022 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:49.647982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:49.683370 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5696b578-cbb6-42bf-9e0d-dd8d4171d022 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:49.685550 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5696b578-cbb6-42bf-9e0d-dd8d4171d022 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:49.793073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 547/1104] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:44:49 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 18 00:44:49.799019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d700f7a-6dc2-49d1-867c-11dcd5a1485d None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:49.892857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:49.896609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d700f7a-6dc2-49d1-867c-11dcd5a1485d None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:49.932944 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8768af0-f2cd-40be-9ecc-99cec2d2a47f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:49.933573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8768af0-f2cd-40be-9ecc-99cec2d2a47f None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:49.933855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8768af0-f2cd-40be-9ecc-99cec2d2a47f None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:49.934131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8768af0-f2cd-40be-9ecc-99cec2d2a47f None admin] PATH_INFO: `/v3/projects/afc8669902004a108a80b47f16395aae` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:49.935352 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8768af0-f2cd-40be-9ecc-99cec2d2a47f None admin] RBAC: Authorizing `identity:get_project(project_id=afc8669902004a108a80b47f16395aae)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:49.936791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8768af0-f2cd-40be-9ecc-99cec2d2a47f None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:49.938442 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 558/1105] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:44:49 2022] GET /identity/v3/projects/afc8669902004a108a80b47f16395aae => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:49.945799 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5696b578-cbb6-42bf-9e0d-dd8d4171d022 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:50.025976 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:50.053761 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5696b578-cbb6-42bf-9e0d-dd8d4171d022 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:50.098971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbbf81ad-b2eb-468b-a62d-fa5bc09f0649 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:50.099586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbf81ad-b2eb-468b-a62d-fa5bc09f0649 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:50.099864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbf81ad-b2eb-468b-a62d-fa5bc09f0649 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:50.100138 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbf81ad-b2eb-468b-a62d-fa5bc09f0649 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:50.100876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbbf81ad-b2eb-468b-a62d-fa5bc09f0649 None admin] RBAC: Adding query filter params (role.id=1e04c9fea65244d28e1e59fc4c64f32f, scope.project.id=afc8669902004a108a80b47f16395aae, user.id=04461da56a8345a18af6ba0fa92cf089) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:50.101187 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbbf81ad-b2eb-468b-a62d-fa5bc09f0649 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:50.102642 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbbf81ad-b2eb-468b-a62d-fa5bc09f0649 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:50.113417 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 548/1106] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:44:49 2022] GET /identity/v3/role_assignments?user.id=04461da56a8345a18af6ba0fa92cf089&scope.project.id=afc8669902004a108a80b47f16395aae&role.id=1e04c9fea65244d28e1e59fc4c64f32f => generated 608 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:52.347196 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f376b2eb-8aa0-4e4b-b98e-0cae3ad0fe42 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:52.347723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f376b2eb-8aa0-4e4b-b98e-0cae3ad0fe42 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:52.347723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f376b2eb-8aa0-4e4b-b98e-0cae3ad0fe42 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:52.349049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 559/1107] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:52.354242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6102ef7f-bf9f-4d2a-9f46-b78f15dcd7cc None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:52.354644 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6102ef7f-bf9f-4d2a-9f46-b78f15dcd7cc None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:52.354814 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6102ef7f-bf9f-4d2a-9f46-b78f15dcd7cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:52.372748 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-6102ef7f-bf9f-4d2a-9f46-b78f15dcd7cc None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:52.373919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-6102ef7f-bf9f-4d2a-9f46-b78f15dcd7cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:52.436821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 549/1108] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:52.710565 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f376b2eb-8aa0-4e4b-b98e-0cae3ad0fe42 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:52.811567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f376b2eb-8aa0-4e4b-b98e-0cae3ad0fe42 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:52.843047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84aabfe6-33b1-4629-a9cd-7b8a9718fc63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:52.843669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84aabfe6-33b1-4629-a9cd-7b8a9718fc63 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:52.843945 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84aabfe6-33b1-4629-a9cd-7b8a9718fc63 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:52.844215 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84aabfe6-33b1-4629-a9cd-7b8a9718fc63 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:52.845303 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84aabfe6-33b1-4629-a9cd-7b8a9718fc63 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:52.846748 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84aabfe6-33b1-4629-a9cd-7b8a9718fc63 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:52.848309 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 560/1109] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:44:52 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:52.854600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6102ef7f-bf9f-4d2a-9f46-b78f15dcd7cc None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:52.918832 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:52.952014 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6102ef7f-bf9f-4d2a-9f46-b78f15dcd7cc None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:52.986229 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc551948-8c6e-4dc9-bfec-f998a4888b97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:52.986798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc551948-8c6e-4dc9-bfec-f998a4888b97 None admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:52.987079 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc551948-8c6e-4dc9-bfec-f998a4888b97 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:52.987347 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc551948-8c6e-4dc9-bfec-f998a4888b97 None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:52.988033 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc551948-8c6e-4dc9-bfec-f998a4888b97 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:52.989494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc551948-8c6e-4dc9-bfec-f998a4888b97 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:53.011832 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 550/1110] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 18 00:44:52 2022] POST /identity/v3/users => generated 286 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:44:55.245128 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae661c45-9874-4601-b569-de3714f3260c None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:55.245656 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae661c45-9874-4601-b569-de3714f3260c None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:55.245656 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae661c45-9874-4601-b569-de3714f3260c None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:55.247046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 561/1111] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:55.252352 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2f60dd-bcb0-4aa0-b3cc-8b249e882d50 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:55.252767 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2f60dd-bcb0-4aa0-b3cc-8b249e882d50 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:55.252906 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2f60dd-bcb0-4aa0-b3cc-8b249e882d50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:55.270851 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-fb2f60dd-bcb0-4aa0-b3cc-8b249e882d50 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:55.272057 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-fb2f60dd-bcb0-4aa0-b3cc-8b249e882d50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:55.337042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 551/1112] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:55 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:55.606125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae661c45-9874-4601-b569-de3714f3260c None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:55.642508 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:55.705865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae661c45-9874-4601-b569-de3714f3260c None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:55.754170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540d5b18-ea7e-4ddc-9192-ede8eca2e7fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:55.754846 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540d5b18-ea7e-4ddc-9192-ede8eca2e7fa None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:55.755129 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540d5b18-ea7e-4ddc-9192-ede8eca2e7fa None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:55.755405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540d5b18-ea7e-4ddc-9192-ede8eca2e7fa None admin] PATH_INFO: `/v3/roles/cf934cf252ce42d5a0445d3e156348d1` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:55.757003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540d5b18-ea7e-4ddc-9192-ede8eca2e7fa None admin] RBAC: Authorizing `identity:get_role(role_id=cf934cf252ce42d5a0445d3e156348d1)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:55.760844 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540d5b18-ea7e-4ddc-9192-ede8eca2e7fa None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:55.761321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 562/1113] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:44:55 2022] GET /identity/v3/roles/cf934cf252ce42d5a0445d3e156348d1 => generated 223 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:55.767252 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2f60dd-bcb0-4aa0-b3cc-8b249e882d50 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:55.827567 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:55.871689 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2f60dd-bcb0-4aa0-b3cc-8b249e882d50 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:55.905028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c5328e-faad-45b9-a7b1-b4eeb1b5d355 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:55.905650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c5328e-faad-45b9-a7b1-b4eeb1b5d355 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:55.905928 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c5328e-faad-45b9-a7b1-b4eeb1b5d355 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:55.906202 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c5328e-faad-45b9-a7b1-b4eeb1b5d355 None admin] PATH_INFO: `/v3/users/efa66a6e3dcd48cdbf2095bfd281abd5` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:55.919225 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c5328e-faad-45b9-a7b1-b4eeb1b5d355 None admin] RBAC: Authorizing `identity:get_user(user_id=efa66a6e3dcd48cdbf2095bfd281abd5)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:55.920913 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c5328e-faad-45b9-a7b1-b4eeb1b5d355 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:55.922839 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 552/1114] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:44:55 2022] GET /identity/v3/users/efa66a6e3dcd48cdbf2095bfd281abd5 => generated 286 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:55.929710 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540d5b18-ea7e-4ddc-9192-ede8eca2e7fa None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:56.025527 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540d5b18-ea7e-4ddc-9192-ede8eca2e7fa None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:56.027805 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:56.072527 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96df6328-a4c0-4b75-aec3-f7d0c8a1d466 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:56.073162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96df6328-a4c0-4b75-aec3-f7d0c8a1d466 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:56.073446 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96df6328-a4c0-4b75-aec3-f7d0c8a1d466 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:56.073724 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96df6328-a4c0-4b75-aec3-f7d0c8a1d466 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:56.151204 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96df6328-a4c0-4b75-aec3-f7d0c8a1d466 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:56.152599 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96df6328-a4c0-4b75-aec3-f7d0c8a1d466 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:56.249124 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 563/1115] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:44:55 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 18 00:44:56.255166 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c5328e-faad-45b9-a7b1-b4eeb1b5d355 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:56.350744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c5328e-faad-45b9-a7b1-b4eeb1b5d355 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:56.363921 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:56.384689 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd50a1d-40bf-42e7-bc4d-89f4b9575b51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:56.385846 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd50a1d-40bf-42e7-bc4d-89f4b9575b51 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:56.386121 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd50a1d-40bf-42e7-bc4d-89f4b9575b51 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:56.386388 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd50a1d-40bf-42e7-bc4d-89f4b9575b51 None admin] PATH_INFO: `/v3/projects/afc8669902004a108a80b47f16395aae` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:56.387528 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd50a1d-40bf-42e7-bc4d-89f4b9575b51 None admin] RBAC: Authorizing `identity:get_project(project_id=afc8669902004a108a80b47f16395aae)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:56.388942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd50a1d-40bf-42e7-bc4d-89f4b9575b51 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:56.390587 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 553/1116] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:44:56 2022] GET /identity/v3/projects/afc8669902004a108a80b47f16395aae => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:56.396858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96df6328-a4c0-4b75-aec3-f7d0c8a1d466 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:56.399995 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:56.497814 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96df6328-a4c0-4b75-aec3-f7d0c8a1d466 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:56.530651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b204d30c-dbb4-43f0-8ee4-9da07c24d1a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:56.531249 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b204d30c-dbb4-43f0-8ee4-9da07c24d1a8 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:56.531523 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b204d30c-dbb4-43f0-8ee4-9da07c24d1a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:56.531790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b204d30c-dbb4-43f0-8ee4-9da07c24d1a8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:56.532502 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b204d30c-dbb4-43f0-8ee4-9da07c24d1a8 None admin] RBAC: Adding query filter params (role.id=cf934cf252ce42d5a0445d3e156348d1, scope.project.id=afc8669902004a108a80b47f16395aae, user.id=efa66a6e3dcd48cdbf2095bfd281abd5) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:44:56.532807 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b204d30c-dbb4-43f0-8ee4-9da07c24d1a8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:56.534215 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b204d30c-dbb4-43f0-8ee4-9da07c24d1a8 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:56.539385 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 564/1117] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:44:56 2022] GET /identity/v3/role_assignments?user.id=efa66a6e3dcd48cdbf2095bfd281abd5&scope.project.id=afc8669902004a108a80b47f16395aae&role.id=cf934cf252ce42d5a0445d3e156348d1 => generated 259 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:58.685778 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f547293-b949-4859-aed0-c93841f2c8e4 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:58.686650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f547293-b949-4859-aed0-c93841f2c8e4 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:58.686650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f547293-b949-4859-aed0-c93841f2c8e4 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:58.687670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 554/1118] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:44:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:44:58.692968 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5514fec7-08b2-43cd-9856-780089d78287 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:58.693419 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5514fec7-08b2-43cd-9856-780089d78287 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:58.693510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5514fec7-08b2-43cd-9856-780089d78287 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:58.711740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-5514fec7-08b2-43cd-9856-780089d78287 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:44:58.712963 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-5514fec7-08b2-43cd-9856-780089d78287 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:44:58.772286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:58.778302 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 565/1119] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:44:58 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:44:59.047417 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f547293-b949-4859-aed0-c93841f2c8e4 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:59.147241 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f547293-b949-4859-aed0-c93841f2c8e4 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:59.180756 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c8de6b-149b-4c00-8b5e-3218dd141325 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:59.181377 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c8de6b-149b-4c00-8b5e-3218dd141325 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:59.181711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c8de6b-149b-4c00-8b5e-3218dd141325 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:59.181986 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c8de6b-149b-4c00-8b5e-3218dd141325 None admin] PATH_INFO: `/v3/roles/cf934cf252ce42d5a0445d3e156348d1` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:59.183368 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c8de6b-149b-4c00-8b5e-3218dd141325 None admin] RBAC: Authorizing `identity:get_role(role_id=cf934cf252ce42d5a0445d3e156348d1)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:59.184721 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c8de6b-149b-4c00-8b5e-3218dd141325 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:59.186035 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 555/1120] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:44:59 2022] GET /identity/v3/roles/cf934cf252ce42d5a0445d3e156348d1 => generated 223 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:59.192025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5514fec7-08b2-43cd-9856-780089d78287 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:59.287660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5514fec7-08b2-43cd-9856-780089d78287 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:59.320166 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77047e1-c52b-4cd2-879c-f991003517be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:59.320796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77047e1-c52b-4cd2-879c-f991003517be None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:59.321075 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77047e1-c52b-4cd2-879c-f991003517be None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:59.321348 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77047e1-c52b-4cd2-879c-f991003517be None admin] PATH_INFO: `/v3/users/efa66a6e3dcd48cdbf2095bfd281abd5` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:59.323049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77047e1-c52b-4cd2-879c-f991003517be None admin] RBAC: Authorizing `identity:get_user(user_id=efa66a6e3dcd48cdbf2095bfd281abd5)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:59.324481 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77047e1-c52b-4cd2-879c-f991003517be None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:59.326063 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 566/1121] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:44:59 2022] GET /identity/v3/users/efa66a6e3dcd48cdbf2095bfd281abd5 => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:59.332041 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c8de6b-149b-4c00-8b5e-3218dd141325 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:59.379807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:59.428955 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c8de6b-149b-4c00-8b5e-3218dd141325 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:59.464994 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231061a8-4ff7-43b3-bff8-4a23d0680e7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:59.465627 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231061a8-4ff7-43b3-bff8-4a23d0680e7a None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:59.465907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231061a8-4ff7-43b3-bff8-4a23d0680e7a None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:59.466181 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231061a8-4ff7-43b3-bff8-4a23d0680e7a None admin] PATH_INFO: `/v3/projects/afc8669902004a108a80b47f16395aae` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:59.467315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-231061a8-4ff7-43b3-bff8-4a23d0680e7a None admin] RBAC: Authorizing `identity:get_project(project_id=afc8669902004a108a80b47f16395aae)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:59.468765 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-231061a8-4ff7-43b3-bff8-4a23d0680e7a None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:59.470518 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 556/1122] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:44:59 2022] GET /identity/v3/projects/afc8669902004a108a80b47f16395aae => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:44:59.476470 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77047e1-c52b-4cd2-879c-f991003517be None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:44:59.544646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:44:59.579975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77047e1-c52b-4cd2-879c-f991003517be None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:44:59.612836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0e6cbd-470b-4f61-aab5-34d6e28bf711 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:44:59.613482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0e6cbd-470b-4f61-aab5-34d6e28bf711 None admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:44:59.613758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0e6cbd-470b-4f61-aab5-34d6e28bf711 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:44:59.614025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0e6cbd-470b-4f61-aab5-34d6e28bf711 None admin] PATH_INFO: `/v3/projects/afc8669902004a108a80b47f16395aae/users/efa66a6e3dcd48cdbf2095bfd281abd5/roles/cf934cf252ce42d5a0445d3e156348d1` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:44:59.616143 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0e6cbd-470b-4f61-aab5-34d6e28bf711 None admin] RBAC: Authorizing `identity:create_grant(project_id=afc8669902004a108a80b47f16395aae, user_id=efa66a6e3dcd48cdbf2095bfd281abd5, role_id=cf934cf252ce42d5a0445d3e156348d1)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:44:59.617665 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0e6cbd-470b-4f61-aab5-34d6e28bf711 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:44:59.630694 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 567/1123] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:44:59 2022] PUT /identity/v3/projects/afc8669902004a108a80b47f16395aae/users/efa66a6e3dcd48cdbf2095bfd281abd5/roles/cf934cf252ce42d5a0445d3e156348d1 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:45:01.771825 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4464beba-e159-4bf5-b161-561ccb54bf23 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:01.772370 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4464beba-e159-4bf5-b161-561ccb54bf23 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:01.772470 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4464beba-e159-4bf5-b161-561ccb54bf23 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:01.773964 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 557/1124] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:45:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:45:01.779571 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80d3e46-0bc2-4a0e-bb53-86f4ed213dc1 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:01.779946 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80d3e46-0bc2-4a0e-bb53-86f4ed213dc1 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:01.780116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80d3e46-0bc2-4a0e-bb53-86f4ed213dc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:01.798140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-c80d3e46-0bc2-4a0e-bb53-86f4ed213dc1 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:45:01.799330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c80d3e46-0bc2-4a0e-bb53-86f4ed213dc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:45:01.862226 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 568/1125] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:45:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:45:02.131960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4464beba-e159-4bf5-b161-561ccb54bf23 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:02.228293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4464beba-e159-4bf5-b161-561ccb54bf23 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:02.245010 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:02.262242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0bf3a2-99ac-47d7-8787-43f53632ef48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:02.262891 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0bf3a2-99ac-47d7-8787-43f53632ef48 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:02.263164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0bf3a2-99ac-47d7-8787-43f53632ef48 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:02.263433 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0bf3a2-99ac-47d7-8787-43f53632ef48 None admin] PATH_INFO: `/v3/roles/cf934cf252ce42d5a0445d3e156348d1` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:02.264794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0bf3a2-99ac-47d7-8787-43f53632ef48 None admin] RBAC: Authorizing `identity:get_role(role_id=cf934cf252ce42d5a0445d3e156348d1)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:02.266144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0bf3a2-99ac-47d7-8787-43f53632ef48 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:02.267481 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 558/1126] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:45:02 2022] GET /identity/v3/roles/cf934cf252ce42d5a0445d3e156348d1 => generated 223 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:02.273373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80d3e46-0bc2-4a0e-bb53-86f4ed213dc1 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:02.326707 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:02.370273 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80d3e46-0bc2-4a0e-bb53-86f4ed213dc1 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:02.403392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ccf4ca-7562-4315-bd09-39331091ffd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:02.403935 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ccf4ca-7562-4315-bd09-39331091ffd7 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:02.404211 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ccf4ca-7562-4315-bd09-39331091ffd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:02.404479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ccf4ca-7562-4315-bd09-39331091ffd7 None admin] PATH_INFO: `/v3/users/efa66a6e3dcd48cdbf2095bfd281abd5` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:02.405588 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ccf4ca-7562-4315-bd09-39331091ffd7 None admin] RBAC: Authorizing `identity:get_user(user_id=efa66a6e3dcd48cdbf2095bfd281abd5)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:02.407047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ccf4ca-7562-4315-bd09-39331091ffd7 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:02.408609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 569/1127] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:45:02 2022] GET /identity/v3/users/efa66a6e3dcd48cdbf2095bfd281abd5 => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:02.414513 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0bf3a2-99ac-47d7-8787-43f53632ef48 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:02.517378 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0bf3a2-99ac-47d7-8787-43f53632ef48 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:02.554060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109aee5c-d746-4d50-a5b0-ddce695712d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:02.554690 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109aee5c-d746-4d50-a5b0-ddce695712d1 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:02.554966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109aee5c-d746-4d50-a5b0-ddce695712d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:02.555232 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109aee5c-d746-4d50-a5b0-ddce695712d1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:02.609119 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:02.624480 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109aee5c-d746-4d50-a5b0-ddce695712d1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:02.625912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109aee5c-d746-4d50-a5b0-ddce695712d1 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:02.720011 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 559/1128] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:45:02 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 18 00:45:02.725833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ccf4ca-7562-4315-bd09-39331091ffd7 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:02.820564 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ccf4ca-7562-4315-bd09-39331091ffd7 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:02.843632 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:02.854400 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b450afe2-ebf2-4671-a9a3-e38c71aff1fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:02.855053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b450afe2-ebf2-4671-a9a3-e38c71aff1fa None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:02.855334 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b450afe2-ebf2-4671-a9a3-e38c71aff1fa None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:02.855609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b450afe2-ebf2-4671-a9a3-e38c71aff1fa None admin] PATH_INFO: `/v3/projects/afc8669902004a108a80b47f16395aae` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:02.856739 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b450afe2-ebf2-4671-a9a3-e38c71aff1fa None admin] RBAC: Authorizing `identity:get_project(project_id=afc8669902004a108a80b47f16395aae)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:02.858172 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b450afe2-ebf2-4671-a9a3-e38c71aff1fa None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:02.859843 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 570/1129] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:45:02 2022] GET /identity/v3/projects/afc8669902004a108a80b47f16395aae => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:02.865977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109aee5c-d746-4d50-a5b0-ddce695712d1 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:02.947719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:02.964700 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109aee5c-d746-4d50-a5b0-ddce695712d1 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:02.998545 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb854252-d78f-4b77-a87e-673ebb724773 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:02.999080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb854252-d78f-4b77-a87e-673ebb724773 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:02.999361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb854252-d78f-4b77-a87e-673ebb724773 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:02.999633 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb854252-d78f-4b77-a87e-673ebb724773 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:03.000363 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb854252-d78f-4b77-a87e-673ebb724773 None admin] RBAC: Adding query filter params (role.id=cf934cf252ce42d5a0445d3e156348d1, scope.project.id=afc8669902004a108a80b47f16395aae, user.id=efa66a6e3dcd48cdbf2095bfd281abd5) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:45:03.000670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb854252-d78f-4b77-a87e-673ebb724773 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:03.002091 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb854252-d78f-4b77-a87e-673ebb724773 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:03.009387 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 560/1130] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:45:02 2022] GET /identity/v3/role_assignments?user.id=efa66a6e3dcd48cdbf2095bfd281abd5&scope.project.id=afc8669902004a108a80b47f16395aae&role.id=cf934cf252ce42d5a0445d3e156348d1 => generated 608 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:05.241899 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e398ff53-77ef-4ea3-8de9-e16d23b58687 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:05.242510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e398ff53-77ef-4ea3-8de9-e16d23b58687 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:05.242510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e398ff53-77ef-4ea3-8de9-e16d23b58687 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:05.243830 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 571/1131] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:45:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:45:05.249806 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72d2d45-ccf8-41f8-9ea5-d4fdd402b6a2 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:05.250202 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72d2d45-ccf8-41f8-9ea5-d4fdd402b6a2 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:05.250378 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72d2d45-ccf8-41f8-9ea5-d4fdd402b6a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:05.268765 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-a72d2d45-ccf8-41f8-9ea5-d4fdd402b6a2 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:45:05.270093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-a72d2d45-ccf8-41f8-9ea5-d4fdd402b6a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:45:05.337495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 561/1132] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:45:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:45:05.628702 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e398ff53-77ef-4ea3-8de9-e16d23b58687 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:05.726041 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e398ff53-77ef-4ea3-8de9-e16d23b58687 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:05.759278 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75c9bc2-3b7e-4788-a7cd-1317eed6c894 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:05.760423 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75c9bc2-3b7e-4788-a7cd-1317eed6c894 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:05.760699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75c9bc2-3b7e-4788-a7cd-1317eed6c894 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:05.760967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75c9bc2-3b7e-4788-a7cd-1317eed6c894 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:05.762094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75c9bc2-3b7e-4788-a7cd-1317eed6c894 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:05.763567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75c9bc2-3b7e-4788-a7cd-1317eed6c894 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:05.765128 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 572/1133] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:45:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:05.770943 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72d2d45-ccf8-41f8-9ea5-d4fdd402b6a2 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:05.843572 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:05.864614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72d2d45-ccf8-41f8-9ea5-d4fdd402b6a2 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:05.897825 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed79f24-9d58-42b5-8716-15a0a2fd5024 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:05.898424 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed79f24-9d58-42b5-8716-15a0a2fd5024 None admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:05.898729 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed79f24-9d58-42b5-8716-15a0a2fd5024 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:05.898997 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed79f24-9d58-42b5-8716-15a0a2fd5024 None admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:05.899636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed79f24-9d58-42b5-8716-15a0a2fd5024 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:05.901597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed79f24-9d58-42b5-8716-15a0a2fd5024 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:05.931141 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 562/1134] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 18 00:45:05 2022] POST /identity/v3/projects => generated 311 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:45:08.121403 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a643fe9-8b09-4e7d-a23a-e5d6f98dd0b8 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:08.121950 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a643fe9-8b09-4e7d-a23a-e5d6f98dd0b8 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:08.122062 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a643fe9-8b09-4e7d-a23a-e5d6f98dd0b8 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:08.123346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 573/1135] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:45:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:45:08.128686 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcb7ccd-e124-43ee-b4f3-91b70be65c89 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:08.129072 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcb7ccd-e124-43ee-b4f3-91b70be65c89 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:08.129223 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcb7ccd-e124-43ee-b4f3-91b70be65c89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:08.147246 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-fbcb7ccd-e124-43ee-b4f3-91b70be65c89 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:45:08.148422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-fbcb7ccd-e124-43ee-b4f3-91b70be65c89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:45:08.212688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 563/1136] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:45:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:45:08.480165 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a643fe9-8b09-4e7d-a23a-e5d6f98dd0b8 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:08.540763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:08.582368 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a643fe9-8b09-4e7d-a23a-e5d6f98dd0b8 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:08.615239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e7d6d7-dc20-4b59-bf11-2526c357f410 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:08.615860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e7d6d7-dc20-4b59-bf11-2526c357f410 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:08.616134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e7d6d7-dc20-4b59-bf11-2526c357f410 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:08.616399 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e7d6d7-dc20-4b59-bf11-2526c357f410 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:08.617482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e7d6d7-dc20-4b59-bf11-2526c357f410 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:08.618980 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e7d6d7-dc20-4b59-bf11-2526c357f410 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:08.621039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 574/1137] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:45:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:08.627021 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcb7ccd-e124-43ee-b4f3-91b70be65c89 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:08.683472 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:08.722472 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcb7ccd-e124-43ee-b4f3-91b70be65c89 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:08.755879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf68346f-fbdc-4be5-91ec-4e9d90ff8d9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:08.756405 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf68346f-fbdc-4be5-91ec-4e9d90ff8d9e None admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:08.756680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf68346f-fbdc-4be5-91ec-4e9d90ff8d9e None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:08.756948 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf68346f-fbdc-4be5-91ec-4e9d90ff8d9e None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:08.757585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf68346f-fbdc-4be5-91ec-4e9d90ff8d9e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:08.759029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf68346f-fbdc-4be5-91ec-4e9d90ff8d9e None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:08.779926 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 564/1138] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 18 00:45:08 2022] POST /identity/v3/users => generated 286 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:45:10.997500 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7e3176-86f5-4436-9119-0f860dae0fa2 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:10.998073 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7e3176-86f5-4436-9119-0f860dae0fa2 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:10.998073 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7e3176-86f5-4436-9119-0f860dae0fa2 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:10.999406 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 575/1139] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:45:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:45:11.004597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1d8a12-bcda-42b2-9793-88775e65efca None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:11.005001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1d8a12-bcda-42b2-9793-88775e65efca None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:11.005181 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1d8a12-bcda-42b2-9793-88775e65efca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:11.023251 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-ad1d8a12-bcda-42b2-9793-88775e65efca None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:45:11.024426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ad1d8a12-bcda-42b2-9793-88775e65efca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:45:11.091767 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 565/1140] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:45:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:45:11.360873 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7e3176-86f5-4436-9119-0f860dae0fa2 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:11.462834 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7e3176-86f5-4436-9119-0f860dae0fa2 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:11.488429 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:11.497195 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cc5756-7128-4587-9fd6-f66014801ad3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:11.497820 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cc5756-7128-4587-9fd6-f66014801ad3 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:11.498098 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cc5756-7128-4587-9fd6-f66014801ad3 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:11.498371 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cc5756-7128-4587-9fd6-f66014801ad3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:11.508303 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6cc5756-7128-4587-9fd6-f66014801ad3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:45:11.513717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-c6cc5756-7128-4587-9fd6-f66014801ad3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:45:11.513717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:45:11.513717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:45:11.513717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:45:11.513717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:45:11.513717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:45:11.513717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:45:11.513717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:45:11.513717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:45:11.513717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:45:11.513717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:45:11.513717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:45:11.513717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:45:11.513717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:45:11.513717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:45:11.513717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:45:11.514931 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:45:11.514931 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:45:11.514931 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:45:11.514931 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:45:11.514931 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:45:11.514931 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:45:11.514931 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:45:11.514931 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:45:11.514931 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:45:11.514931 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:45:11.514931 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:45:11.514931 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:45:11.514931 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:45:11.514931 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:45:11.514931 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:45:11.514931 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:45:11.515857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:45:11.515857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:45:11.515857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:45:11.515857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:45:11.515857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:45:11.515857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:45:11.515857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:45:11.515857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:45:11.515857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:45:11.515857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:45:11.515857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:45:11.515857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:45:11.515857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 576/1141] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:45:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:45:11.521654 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1d8a12-bcda-42b2-9793-88775e65efca None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:11.571257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:11.625492 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1d8a12-bcda-42b2-9793-88775e65efca None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:11.660283 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c67ed76-0c5b-427f-880f-212a10a036ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:11.660882 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c67ed76-0c5b-427f-880f-212a10a036ed None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:11.661157 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c67ed76-0c5b-427f-880f-212a10a036ed None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:11.661425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c67ed76-0c5b-427f-880f-212a10a036ed None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:11.662083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c67ed76-0c5b-427f-880f-212a10a036ed None admin] RBAC: Adding query filter params (name=admin) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:45:11.662380 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c67ed76-0c5b-427f-880f-212a10a036ed None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:11.663751 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c67ed76-0c5b-427f-880f-212a10a036ed None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:11.671377 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 566/1142] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:45:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:11.677516 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cc5756-7128-4587-9fd6-f66014801ad3 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:11.772262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cc5756-7128-4587-9fd6-f66014801ad3 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:11.805081 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b17ddd7-615d-492f-b1a8-3280f9c981e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:11.805707 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b17ddd7-615d-492f-b1a8-3280f9c981e1 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:11.805986 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b17ddd7-615d-492f-b1a8-3280f9c981e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:11.806258 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b17ddd7-615d-492f-b1a8-3280f9c981e1 None admin] PATH_INFO: `/v3/users/069a4599ec3140038785ddec2451b5e0` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:11.819499 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b17ddd7-615d-492f-b1a8-3280f9c981e1 None admin] RBAC: Authorizing `identity:get_user(user_id=069a4599ec3140038785ddec2451b5e0)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:11.820924 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b17ddd7-615d-492f-b1a8-3280f9c981e1 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:11.823068 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 577/1143] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:45:11 2022] GET /identity/v3/users/069a4599ec3140038785ddec2451b5e0 => generated 286 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:11.829133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c67ed76-0c5b-427f-880f-212a10a036ed None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:11.934338 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c67ed76-0c5b-427f-880f-212a10a036ed None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:11.942961 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:11.968546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07625811-abf0-4bc7-96d2-9af08f2cb48f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:11.969154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07625811-abf0-4bc7-96d2-9af08f2cb48f None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:11.969433 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07625811-abf0-4bc7-96d2-9af08f2cb48f None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:11.969706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07625811-abf0-4bc7-96d2-9af08f2cb48f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:12.034467 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07625811-abf0-4bc7-96d2-9af08f2cb48f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:12.035916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07625811-abf0-4bc7-96d2-9af08f2cb48f None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:12.129401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 567/1144] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:45:11 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 18 00:45:12.135966 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b17ddd7-615d-492f-b1a8-3280f9c981e1 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:12.180343 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:12.231950 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b17ddd7-615d-492f-b1a8-3280f9c981e1 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:12.266835 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a43c5e-ee2b-41d1-beb9-e71cbd9b6d53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:12.267482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a43c5e-ee2b-41d1-beb9-e71cbd9b6d53 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:12.267761 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a43c5e-ee2b-41d1-beb9-e71cbd9b6d53 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:12.268034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a43c5e-ee2b-41d1-beb9-e71cbd9b6d53 None admin] PATH_INFO: `/v3/projects/8ca7b2af128d4651834e59db5402f3df` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:12.269213 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a43c5e-ee2b-41d1-beb9-e71cbd9b6d53 None admin] RBAC: Authorizing `identity:get_project(project_id=8ca7b2af128d4651834e59db5402f3df)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:12.270724 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a43c5e-ee2b-41d1-beb9-e71cbd9b6d53 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:12.272447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 578/1145] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:45:12 2022] GET /identity/v3/projects/8ca7b2af128d4651834e59db5402f3df => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:12.278918 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07625811-abf0-4bc7-96d2-9af08f2cb48f None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:12.289177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:12.379137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07625811-abf0-4bc7-96d2-9af08f2cb48f None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:12.412975 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b55c21-ef2a-4e91-81c4-86e2fcb6b030 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:12.413579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b55c21-ef2a-4e91-81c4-86e2fcb6b030 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:12.413855 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b55c21-ef2a-4e91-81c4-86e2fcb6b030 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:12.414125 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b55c21-ef2a-4e91-81c4-86e2fcb6b030 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:12.414880 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b55c21-ef2a-4e91-81c4-86e2fcb6b030 None admin] RBAC: Adding query filter params (role.id=1e04c9fea65244d28e1e59fc4c64f32f, scope.project.id=8ca7b2af128d4651834e59db5402f3df, user.id=069a4599ec3140038785ddec2451b5e0) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:45:12.415188 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b55c21-ef2a-4e91-81c4-86e2fcb6b030 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:12.417202 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b55c21-ef2a-4e91-81c4-86e2fcb6b030 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:12.422822 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 568/1146] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:45:12 2022] GET /identity/v3/role_assignments?user.id=069a4599ec3140038785ddec2451b5e0&scope.project.id=8ca7b2af128d4651834e59db5402f3df&role.id=1e04c9fea65244d28e1e59fc4c64f32f => generated 259 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:14.574916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d137934b-e24d-4889-b512-36e554b3e8ef None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:14.575438 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d137934b-e24d-4889-b512-36e554b3e8ef None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:14.575546 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d137934b-e24d-4889-b512-36e554b3e8ef None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:14.576806 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 579/1147] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:45:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:45:14.581924 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67103be-71a7-4b62-b1f4-c1ddd0f7779f None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:14.582302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67103be-71a7-4b62-b1f4-c1ddd0f7779f None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:14.582504 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67103be-71a7-4b62-b1f4-c1ddd0f7779f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:14.600769 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-b67103be-71a7-4b62-b1f4-c1ddd0f7779f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:45:14.601968 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-b67103be-71a7-4b62-b1f4-c1ddd0f7779f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:45:14.667021 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 569/1148] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:45:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:45:14.942642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d137934b-e24d-4889-b512-36e554b3e8ef None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:15.040821 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d137934b-e24d-4889-b512-36e554b3e8ef None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:15.054556 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:15.075099 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddeefda8-f7d6-40a0-93fb-a530624dc5de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:15.075723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeefda8-f7d6-40a0-93fb-a530624dc5de None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:15.075994 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeefda8-f7d6-40a0-93fb-a530624dc5de None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:15.076264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeefda8-f7d6-40a0-93fb-a530624dc5de None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:15.085856 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddeefda8-f7d6-40a0-93fb-a530624dc5de None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:45:15.090922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-ddeefda8-f7d6-40a0-93fb-a530624dc5de None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:45:15.090922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:45:15.090922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:45:15.090922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:45:15.090922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:45:15.090922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:45:15.090922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:45:15.090922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:45:15.090922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:45:15.090922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:45:15.090922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:45:15.090922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:45:15.090922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:45:15.090922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:45:15.090922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:45:15.090922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:45:15.092023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:45:15.092023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:45:15.092023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:45:15.092023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:45:15.092023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:45:15.092023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:45:15.092023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:45:15.092023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:45:15.092023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:45:15.092023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:45:15.092023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:45:15.092023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:45:15.092023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:45:15.092023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:45:15.092023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:45:15.092023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:45:15.092962 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:45:15.092962 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:45:15.092962 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:45:15.092962 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:45:15.092962 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:45:15.092962 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:45:15.092962 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:45:15.092962 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:45:15.092962 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:45:15.092962 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:45:15.092962 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:45:15.092962 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:45:15.092962 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 580/1149] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:45:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:45:15.098163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67103be-71a7-4b62-b1f4-c1ddd0f7779f None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:15.100332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:15.193911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67103be-71a7-4b62-b1f4-c1ddd0f7779f None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:15.225870 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079567ee-eac3-4d40-9005-ebf961c8a007 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:15.226479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079567ee-eac3-4d40-9005-ebf961c8a007 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:15.226754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079567ee-eac3-4d40-9005-ebf961c8a007 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:15.227022 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079567ee-eac3-4d40-9005-ebf961c8a007 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:15.227671 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079567ee-eac3-4d40-9005-ebf961c8a007 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:45:15.227964 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079567ee-eac3-4d40-9005-ebf961c8a007 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:15.229341 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079567ee-eac3-4d40-9005-ebf961c8a007 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:15.237025 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 570/1150] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:45:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:15.242910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddeefda8-f7d6-40a0-93fb-a530624dc5de None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:15.339042 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddeefda8-f7d6-40a0-93fb-a530624dc5de None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:15.371458 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d897af6d-82df-4ff0-b017-51ac3b6a8822 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:15.372075 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d897af6d-82df-4ff0-b017-51ac3b6a8822 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:15.372349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d897af6d-82df-4ff0-b017-51ac3b6a8822 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:15.372621 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d897af6d-82df-4ff0-b017-51ac3b6a8822 None admin] PATH_INFO: `/v3/users/069a4599ec3140038785ddec2451b5e0` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:15.373714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d897af6d-82df-4ff0-b017-51ac3b6a8822 None admin] RBAC: Authorizing `identity:get_user(user_id=069a4599ec3140038785ddec2451b5e0)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:15.375165 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d897af6d-82df-4ff0-b017-51ac3b6a8822 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:15.376725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 581/1151] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:45:15 2022] GET /identity/v3/users/069a4599ec3140038785ddec2451b5e0 => generated 286 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:15.382684 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079567ee-eac3-4d40-9005-ebf961c8a007 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:15.444413 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:15.481725 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079567ee-eac3-4d40-9005-ebf961c8a007 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:15.514981 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a36dcec-9178-435b-8b6c-8ffb0ffb5e8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:15.515610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a36dcec-9178-435b-8b6c-8ffb0ffb5e8e None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:15.515883 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a36dcec-9178-435b-8b6c-8ffb0ffb5e8e None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:15.516151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a36dcec-9178-435b-8b6c-8ffb0ffb5e8e None admin] PATH_INFO: `/v3/projects/8ca7b2af128d4651834e59db5402f3df` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:15.517255 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a36dcec-9178-435b-8b6c-8ffb0ffb5e8e None admin] RBAC: Authorizing `identity:get_project(project_id=8ca7b2af128d4651834e59db5402f3df)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:15.518713 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a36dcec-9178-435b-8b6c-8ffb0ffb5e8e None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:15.520334 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 571/1152] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:45:15 2022] GET /identity/v3/projects/8ca7b2af128d4651834e59db5402f3df => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:15.526889 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d897af6d-82df-4ff0-b017-51ac3b6a8822 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:15.569999 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:15.625141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d897af6d-82df-4ff0-b017-51ac3b6a8822 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:15.658279 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5def3bb0-39fc-4211-af59-9fd2b8fb396d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:15.658985 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5def3bb0-39fc-4211-af59-9fd2b8fb396d None admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:15.659262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5def3bb0-39fc-4211-af59-9fd2b8fb396d None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:15.659537 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5def3bb0-39fc-4211-af59-9fd2b8fb396d None admin] PATH_INFO: `/v3/projects/8ca7b2af128d4651834e59db5402f3df/users/069a4599ec3140038785ddec2451b5e0/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:15.661433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5def3bb0-39fc-4211-af59-9fd2b8fb396d None admin] RBAC: Authorizing `identity:create_grant(project_id=8ca7b2af128d4651834e59db5402f3df, user_id=069a4599ec3140038785ddec2451b5e0, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:15.662894 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5def3bb0-39fc-4211-af59-9fd2b8fb396d None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:15.675598 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 582/1153] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:45:15 2022] PUT /identity/v3/projects/8ca7b2af128d4651834e59db5402f3df/users/069a4599ec3140038785ddec2451b5e0/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:45:17.805391 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1f5587-5179-402c-a26a-94407618452b None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:17.805910 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1f5587-5179-402c-a26a-94407618452b None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:17.805910 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1f5587-5179-402c-a26a-94407618452b None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:17.807374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 572/1154] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:45:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:45:17.812534 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c57df24-87f7-4b8a-a3fa-12b97cae7d38 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:17.812976 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c57df24-87f7-4b8a-a3fa-12b97cae7d38 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:17.813081 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c57df24-87f7-4b8a-a3fa-12b97cae7d38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:17.832252 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-9c57df24-87f7-4b8a-a3fa-12b97cae7d38 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:45:17.833465 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-9c57df24-87f7-4b8a-a3fa-12b97cae7d38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:45:17.898781 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 583/1155] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:45:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:45:18.172111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1f5587-5179-402c-a26a-94407618452b None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:18.271467 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1f5587-5179-402c-a26a-94407618452b None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:18.299346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:18.304926 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6779d7-386d-44fd-a0fd-54932fbc7d58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:18.305545 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6779d7-386d-44fd-a0fd-54932fbc7d58 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:18.305819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6779d7-386d-44fd-a0fd-54932fbc7d58 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:18.306083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6779d7-386d-44fd-a0fd-54932fbc7d58 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:18.316195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6779d7-386d-44fd-a0fd-54932fbc7d58 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:45:18.320917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-ed6779d7-386d-44fd-a0fd-54932fbc7d58 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:45:18.320917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:45:18.320917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:45:18.320917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:45:18.320917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:45:18.320917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:45:18.320917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:45:18.320917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:45:18.320917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:45:18.320917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:45:18.320917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:45:18.320917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:45:18.320917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:45:18.320917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:45:18.320917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:45:18.320917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:45:18.321867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:45:18.321867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:45:18.321867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:45:18.321867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:45:18.321867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:45:18.321867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:45:18.321867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:45:18.321867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:45:18.321867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:45:18.321867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:45:18.321867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:45:18.321867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:45:18.321867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:45:18.321867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:45:18.321867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:45:18.321867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:45:18.322960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:45:18.322960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:45:18.322960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:45:18.322960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:45:18.322960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:45:18.322960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:45:18.322960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:45:18.322960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:45:18.322960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:45:18.322960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:45:18.322960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:45:18.322960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:45:18.322960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 573/1156] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:45:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:45:18.328150 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c57df24-87f7-4b8a-a3fa-12b97cae7d38 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:18.357530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:18.423952 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c57df24-87f7-4b8a-a3fa-12b97cae7d38 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:18.456612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066a035a-0519-40c3-aed5-1f930f0eac31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:18.457210 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066a035a-0519-40c3-aed5-1f930f0eac31 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:18.457483 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066a035a-0519-40c3-aed5-1f930f0eac31 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:18.457750 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066a035a-0519-40c3-aed5-1f930f0eac31 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:18.458402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-066a035a-0519-40c3-aed5-1f930f0eac31 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:45:18.458729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-066a035a-0519-40c3-aed5-1f930f0eac31 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:18.460079 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-066a035a-0519-40c3-aed5-1f930f0eac31 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:18.468316 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 584/1157] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:45:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:18.474409 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6779d7-386d-44fd-a0fd-54932fbc7d58 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:18.570986 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6779d7-386d-44fd-a0fd-54932fbc7d58 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:18.604306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be591d67-0460-49f3-beb6-210b263dc73c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:18.604931 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be591d67-0460-49f3-beb6-210b263dc73c None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:18.605213 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be591d67-0460-49f3-beb6-210b263dc73c None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:18.605488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be591d67-0460-49f3-beb6-210b263dc73c None admin] PATH_INFO: `/v3/users/069a4599ec3140038785ddec2451b5e0` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:18.606639 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be591d67-0460-49f3-beb6-210b263dc73c None admin] RBAC: Authorizing `identity:get_user(user_id=069a4599ec3140038785ddec2451b5e0)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:18.608065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be591d67-0460-49f3-beb6-210b263dc73c None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:18.609638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 574/1158] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:45:18 2022] GET /identity/v3/users/069a4599ec3140038785ddec2451b5e0 => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:18.615528 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066a035a-0519-40c3-aed5-1f930f0eac31 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:18.705999 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:18.712603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066a035a-0519-40c3-aed5-1f930f0eac31 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:18.745567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5405bf98-0dcb-4dfe-8fed-f4d7bd7ba0cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:18.746173 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5405bf98-0dcb-4dfe-8fed-f4d7bd7ba0cf None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:18.746452 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5405bf98-0dcb-4dfe-8fed-f4d7bd7ba0cf None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:18.746754 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5405bf98-0dcb-4dfe-8fed-f4d7bd7ba0cf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:18.811327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5405bf98-0dcb-4dfe-8fed-f4d7bd7ba0cf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:18.812767 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5405bf98-0dcb-4dfe-8fed-f4d7bd7ba0cf None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:18.898133 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:18.906307 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 585/1159] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:45:18 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 18 00:45:18.912304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be591d67-0460-49f3-beb6-210b263dc73c None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:18.972082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:19.016079 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be591d67-0460-49f3-beb6-210b263dc73c None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:19.049181 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32152135-4956-4784-83e5-2158c645035f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:19.049795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32152135-4956-4784-83e5-2158c645035f None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:19.050069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32152135-4956-4784-83e5-2158c645035f None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:19.050336 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32152135-4956-4784-83e5-2158c645035f None admin] PATH_INFO: `/v3/projects/8ca7b2af128d4651834e59db5402f3df` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:19.051454 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32152135-4956-4784-83e5-2158c645035f None admin] RBAC: Authorizing `identity:get_project(project_id=8ca7b2af128d4651834e59db5402f3df)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:19.053119 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32152135-4956-4784-83e5-2158c645035f None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:19.054764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 575/1160] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:45:18 2022] GET /identity/v3/projects/8ca7b2af128d4651834e59db5402f3df => generated 311 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:19.060948 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5405bf98-0dcb-4dfe-8fed-f4d7bd7ba0cf None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:19.158989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5405bf98-0dcb-4dfe-8fed-f4d7bd7ba0cf None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:19.191650 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60e402a-ca0b-4be0-9d91-8cd4119af525 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:19.192243 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60e402a-ca0b-4be0-9d91-8cd4119af525 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:19.192522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60e402a-ca0b-4be0-9d91-8cd4119af525 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:19.192789 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60e402a-ca0b-4be0-9d91-8cd4119af525 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:19.193512 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60e402a-ca0b-4be0-9d91-8cd4119af525 None admin] RBAC: Adding query filter params (role.id=1e04c9fea65244d28e1e59fc4c64f32f, scope.project.id=8ca7b2af128d4651834e59db5402f3df, user.id=069a4599ec3140038785ddec2451b5e0) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:45:19.193814 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60e402a-ca0b-4be0-9d91-8cd4119af525 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:19.195262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60e402a-ca0b-4be0-9d91-8cd4119af525 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:19.201463 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 586/1161] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:45:19 2022] GET /identity/v3/role_assignments?user.id=069a4599ec3140038785ddec2451b5e0&scope.project.id=8ca7b2af128d4651834e59db5402f3df&role.id=1e04c9fea65244d28e1e59fc4c64f32f => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:21.381172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce881765-863c-4a09-82aa-ea26e3c66e72 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:21.381693 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce881765-863c-4a09-82aa-ea26e3c66e72 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:21.381693 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce881765-863c-4a09-82aa-ea26e3c66e72 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:21.383071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 576/1162] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:45:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:45:21.388744 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8592d2f-bdc3-4f00-82a6-8f9dfd0a25d4 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:21.389155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8592d2f-bdc3-4f00-82a6-8f9dfd0a25d4 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:21.389323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8592d2f-bdc3-4f00-82a6-8f9dfd0a25d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:21.407772 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-c8592d2f-bdc3-4f00-82a6-8f9dfd0a25d4 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:45:21.408955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c8592d2f-bdc3-4f00-82a6-8f9dfd0a25d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:45:21.461160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:21.475055 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 587/1163] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:45:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:45:21.740907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce881765-863c-4a09-82aa-ea26e3c66e72 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:21.843199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce881765-863c-4a09-82aa-ea26e3c66e72 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:21.868691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:21.876633 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36741195-c8c3-481f-9a90-05e8d64ef590 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:21.877236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36741195-c8c3-481f-9a90-05e8d64ef590 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:21.877509 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36741195-c8c3-481f-9a90-05e8d64ef590 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:21.877777 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36741195-c8c3-481f-9a90-05e8d64ef590 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:21.946129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36741195-c8c3-481f-9a90-05e8d64ef590 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:21.947603 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36741195-c8c3-481f-9a90-05e8d64ef590 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:22.042996 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 577/1164] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:45:21 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 18 00:45:22.048821 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8592d2f-bdc3-4f00-82a6-8f9dfd0a25d4 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:22.144350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8592d2f-bdc3-4f00-82a6-8f9dfd0a25d4 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:22.178714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11c306d-ac24-4fb5-8264-7c6bf2eea7a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:22.179343 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11c306d-ac24-4fb5-8264-7c6bf2eea7a1 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:22.179622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11c306d-ac24-4fb5-8264-7c6bf2eea7a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:22.179892 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11c306d-ac24-4fb5-8264-7c6bf2eea7a1 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:22.186450 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11c306d-ac24-4fb5-8264-7c6bf2eea7a1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:22.187927 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11c306d-ac24-4fb5-8264-7c6bf2eea7a1 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:22.199740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-a11c306d-ac24-4fb5-8264-7c6bf2eea7a1 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 18 00:45:22.199740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:45:22.199740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Sep 18 00:45:22.199740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Sep 18 00:45:22.199740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:45:22.199740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:45:22.199740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:45:22.199740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:45:22.199740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Sep 18 00:45:22.199740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:45:22.199740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:45:22.199740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:45:22.199740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:45:22.199740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:45:22.199740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:45:22.199740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:45:22.199740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:45:22.201094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:45:22.201094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:45:22.201094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:45:22.201094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:45:22.201094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:45:22.201094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:45:22.201094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:45:22.201094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:45:22.201094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:45:22.201094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:45:22.201094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:45:22.201094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:45:22.201094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:45:22.201094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:45:22.201094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:45:22.201094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:45:22.202040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:45:22.202040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:45:22.202040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:45:22.202040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:45:22.202040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:45:22.202040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:45:22.202040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:45:22.202040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:45:22.202040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:45:22.202040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:45:22.202040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Sep 18 00:45:22.202040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:45:22.202040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 18 00:45:22.202040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:45:22.202040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 588/1165] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Sep 18 00:45:22 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:45:22.207034 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36741195-c8c3-481f-9a90-05e8d64ef590 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:22.234303 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:22.306666 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36741195-c8c3-481f-9a90-05e8d64ef590 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:22.340450 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda98e00-41eb-4671-9c75-15f6a7c7dc58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:22.341048 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda98e00-41eb-4671-9c75-15f6a7c7dc58 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:22.341321 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda98e00-41eb-4671-9c75-15f6a7c7dc58 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:22.341588 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda98e00-41eb-4671-9c75-15f6a7c7dc58 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:22.342230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda98e00-41eb-4671-9c75-15f6a7c7dc58 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:45:22.342553 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda98e00-41eb-4671-9c75-15f6a7c7dc58 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:22.343898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda98e00-41eb-4671-9c75-15f6a7c7dc58 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:22.351023 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 578/1166] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Sep 18 00:45:22 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:24.517276 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce7e762-d4ec-4505-8df9-fd73be924ed0 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:24.517859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce7e762-d4ec-4505-8df9-fd73be924ed0 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:24.517859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce7e762-d4ec-4505-8df9-fd73be924ed0 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:24.519221 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 589/1167] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:45:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:45:24.525116 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038fcfaf-7618-4fd8-a0b2-7a04c4cad4c0 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:24.525481 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038fcfaf-7618-4fd8-a0b2-7a04c4cad4c0 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:24.525750 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038fcfaf-7618-4fd8-a0b2-7a04c4cad4c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:24.544025 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-038fcfaf-7618-4fd8-a0b2-7a04c4cad4c0 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:45:24.545228 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-038fcfaf-7618-4fd8-a0b2-7a04c4cad4c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:45:24.614566 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 579/1168] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:45:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:45:24.882029 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce7e762-d4ec-4505-8df9-fd73be924ed0 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:24.930446 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:24.981159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce7e762-d4ec-4505-8df9-fd73be924ed0 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:25.013135 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3eafd0-0634-4e7d-94a1-e38066d84cf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:25.013731 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3eafd0-0634-4e7d-94a1-e38066d84cf9 None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:25.014007 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3eafd0-0634-4e7d-94a1-e38066d84cf9 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:25.014276 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3eafd0-0634-4e7d-94a1-e38066d84cf9 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:25.014968 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3eafd0-0634-4e7d-94a1-e38066d84cf9 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:25.016325 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3eafd0-0634-4e7d-94a1-e38066d84cf9 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:25.032225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 590/1169] 10.222.0.42 () {60 vars in 1138 bytes} [Sun Sep 18 00:45:24 2022] POST /identity/v3/domains => generated 267 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:45:27.267920 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b4a933-ffe4-4f01-87d0-82a5bdc67dd4 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:27.268443 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b4a933-ffe4-4f01-87d0-82a5bdc67dd4 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:27.268443 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b4a933-ffe4-4f01-87d0-82a5bdc67dd4 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:27.269835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 580/1170] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:45:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:45:27.275571 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9338ca1-333d-4000-948c-1b6b7125024e None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:27.276026 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9338ca1-333d-4000-948c-1b6b7125024e None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:27.276128 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9338ca1-333d-4000-948c-1b6b7125024e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:27.295016 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-c9338ca1-333d-4000-948c-1b6b7125024e None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:45:27.296205 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c9338ca1-333d-4000-948c-1b6b7125024e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:45:27.365966 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 591/1171] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:45:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:45:27.636057 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b4a933-ffe4-4f01-87d0-82a5bdc67dd4 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:27.660279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:27.735302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b4a933-ffe4-4f01-87d0-82a5bdc67dd4 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:27.767144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a934e56-6a9d-451d-a3b3-7d397d9961b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:27.767760 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a934e56-6a9d-451d-a3b3-7d397d9961b0 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:27.768035 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a934e56-6a9d-451d-a3b3-7d397d9961b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:27.768303 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a934e56-6a9d-451d-a3b3-7d397d9961b0 None admin] PATH_INFO: `/v3/domains/c0a160b47ab747babce6034364937953` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:27.778687 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a934e56-6a9d-451d-a3b3-7d397d9961b0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=c0a160b47ab747babce6034364937953)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:27.780052 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a934e56-6a9d-451d-a3b3-7d397d9961b0 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:27.781616 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 581/1172] 10.222.0.42 () {56 vars in 1183 bytes} [Sun Sep 18 00:45:27 2022] GET /identity/v3/domains/c0a160b47ab747babce6034364937953 => generated 267 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:27.787536 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9338ca1-333d-4000-948c-1b6b7125024e None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:27.825280 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:27.882625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9338ca1-333d-4000-948c-1b6b7125024e None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:27.915535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277bca5e-56a8-4d29-b054-0c26f75af69e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:27.916683 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277bca5e-56a8-4d29-b054-0c26f75af69e None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:27.916984 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277bca5e-56a8-4d29-b054-0c26f75af69e None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:27.917239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277bca5e-56a8-4d29-b054-0c26f75af69e None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:27.917893 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277bca5e-56a8-4d29-b054-0c26f75af69e None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:27.919354 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277bca5e-56a8-4d29-b054-0c26f75af69e None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:27.945776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 592/1173] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 18 00:45:27 2022] POST /identity/v3/projects => generated 361 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:45:30.127387 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d75f8d0-a004-461c-8f16-0b656ff4ebc5 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:30.127910 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d75f8d0-a004-461c-8f16-0b656ff4ebc5 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:30.127910 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d75f8d0-a004-461c-8f16-0b656ff4ebc5 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:30.129251 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 582/1174] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:45:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:45:30.134662 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a268dcbb-bae9-412d-966e-76776a25e56f None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:30.135043 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a268dcbb-bae9-412d-966e-76776a25e56f None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:30.135211 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a268dcbb-bae9-412d-966e-76776a25e56f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:30.153337 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-a268dcbb-bae9-412d-966e-76776a25e56f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:45:30.154527 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a268dcbb-bae9-412d-966e-76776a25e56f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:45:30.218470 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 593/1175] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:45:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 18 00:45:30.487440 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d75f8d0-a004-461c-8f16-0b656ff4ebc5 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:30.572617 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:30.591386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d75f8d0-a004-461c-8f16-0b656ff4ebc5 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:30.625578 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ffb474-ab91-43e5-bca4-e9d08bae9f98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:30.626211 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ffb474-ab91-43e5-bca4-e9d08bae9f98 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:30.626517 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ffb474-ab91-43e5-bca4-e9d08bae9f98 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:30.626785 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ffb474-ab91-43e5-bca4-e9d08bae9f98 None admin] PATH_INFO: `/v3/domains/c0a160b47ab747babce6034364937953` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:30.627916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ffb474-ab91-43e5-bca4-e9d08bae9f98 None admin] RBAC: Authorizing `identity:get_domain(domain_id=c0a160b47ab747babce6034364937953)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:30.629346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ffb474-ab91-43e5-bca4-e9d08bae9f98 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:30.630965 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 583/1176] 10.222.0.42 () {56 vars in 1183 bytes} [Sun Sep 18 00:45:30 2022] GET /identity/v3/domains/c0a160b47ab747babce6034364937953 => generated 267 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:30.636793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a268dcbb-bae9-412d-966e-76776a25e56f None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:30.658045 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:30.736470 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a268dcbb-bae9-412d-966e-76776a25e56f None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:30.769573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20649c56-81b0-4f53-8c88-c443a8040d01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:30.770095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20649c56-81b0-4f53-8c88-c443a8040d01 None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:30.770367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20649c56-81b0-4f53-8c88-c443a8040d01 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:30.770660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20649c56-81b0-4f53-8c88-c443a8040d01 None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:30.771311 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20649c56-81b0-4f53-8c88-c443a8040d01 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:30.772716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20649c56-81b0-4f53-8c88-c443a8040d01 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:30.793194 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 594/1177] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 18 00:45:30 2022] POST /identity/v3/users => generated 311 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:45:32.986952 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8ae998-b956-46ff-a2d6-8794f2c0345c None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:32.987478 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8ae998-b956-46ff-a2d6-8794f2c0345c None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:32.987620 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8ae998-b956-46ff-a2d6-8794f2c0345c None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:32.988843 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 584/1178] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:45:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:45:32.995038 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e975482-02d9-422d-9fce-fbf0d532109f None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:32.995465 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e975482-02d9-422d-9fce-fbf0d532109f None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:32.995586 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e975482-02d9-422d-9fce-fbf0d532109f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:33.014149 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-7e975482-02d9-422d-9fce-fbf0d532109f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:45:33.015374 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-7e975482-02d9-422d-9fce-fbf0d532109f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:45:33.079374 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 595/1179] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:45:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:45:33.349459 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8ae998-b956-46ff-a2d6-8794f2c0345c None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:33.450991 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8ae998-b956-46ff-a2d6-8794f2c0345c None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:33.483740 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77629755-2ed1-40b3-b225-20b4ab905ef3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:33.484376 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77629755-2ed1-40b3-b225-20b4ab905ef3 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:33.484650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77629755-2ed1-40b3-b225-20b4ab905ef3 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:33.484918 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77629755-2ed1-40b3-b225-20b4ab905ef3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:33.495349 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77629755-2ed1-40b3-b225-20b4ab905ef3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:45:33.500124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-77629755-2ed1-40b3-b225-20b4ab905ef3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:45:33.500124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:45:33.500124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:45:33.500124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:45:33.500124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:45:33.500124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:45:33.500124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:45:33.500124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:45:33.500124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:45:33.500124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:45:33.500124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:45:33.500124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:45:33.500124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:45:33.500124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:45:33.500124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:45:33.500124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:45:33.501191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:45:33.501191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:45:33.501191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:45:33.501191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:45:33.501191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:45:33.501191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:45:33.501191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:45:33.501191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:45:33.501191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:45:33.501191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:45:33.501191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:45:33.501191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:45:33.501191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:45:33.501191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:45:33.501191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:45:33.501191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:45:33.502230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:45:33.502230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:45:33.502230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:45:33.502230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:45:33.502230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:45:33.502230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:45:33.502230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:45:33.502230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:45:33.502230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:45:33.502230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:45:33.502230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:45:33.502230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:45:33.502230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 585/1180] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:45:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:45:33.507301 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e975482-02d9-422d-9fce-fbf0d532109f None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:33.515811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:33.606215 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e975482-02d9-422d-9fce-fbf0d532109f None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:33.642550 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d7fda6-46ec-478b-b085-15e87092a88a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:33.643148 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d7fda6-46ec-478b-b085-15e87092a88a None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:33.643425 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d7fda6-46ec-478b-b085-15e87092a88a None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:33.643693 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d7fda6-46ec-478b-b085-15e87092a88a None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:33.644349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d7fda6-46ec-478b-b085-15e87092a88a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:45:33.644657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d7fda6-46ec-478b-b085-15e87092a88a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:33.646011 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d7fda6-46ec-478b-b085-15e87092a88a None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:33.654472 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 596/1181] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:45:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:33.660475 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77629755-2ed1-40b3-b225-20b4ab905ef3 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:33.672787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:33.759342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77629755-2ed1-40b3-b225-20b4ab905ef3 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:33.791797 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f3d3f7-d53c-4a42-8456-438f6faee7b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:33.792426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f3d3f7-d53c-4a42-8456-438f6faee7b4 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:33.792703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f3d3f7-d53c-4a42-8456-438f6faee7b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:33.792974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f3d3f7-d53c-4a42-8456-438f6faee7b4 None admin] PATH_INFO: `/v3/users/bb13fb11841c4ebe92382b2834548686` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:33.805911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f3d3f7-d53c-4a42-8456-438f6faee7b4 None admin] RBAC: Authorizing `identity:get_user(user_id=bb13fb11841c4ebe92382b2834548686)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:33.807860 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f3d3f7-d53c-4a42-8456-438f6faee7b4 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:33.809471 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 586/1182] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:45:33 2022] GET /identity/v3/users/bb13fb11841c4ebe92382b2834548686 => generated 311 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:33.815408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d7fda6-46ec-478b-b085-15e87092a88a None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:33.879452 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:33.912602 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d7fda6-46ec-478b-b085-15e87092a88a None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:33.944966 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe51990-9f1c-441f-bbd5-1e814b25fda0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:33.945568 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe51990-9f1c-441f-bbd5-1e814b25fda0 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:33.945845 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe51990-9f1c-441f-bbd5-1e814b25fda0 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:33.946115 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe51990-9f1c-441f-bbd5-1e814b25fda0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:34.010446 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe51990-9f1c-441f-bbd5-1e814b25fda0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:34.011925 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe51990-9f1c-441f-bbd5-1e814b25fda0 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:34.073685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:34.107758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 597/1183] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:45:33 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 18 00:45:34.113704 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f3d3f7-d53c-4a42-8456-438f6faee7b4 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:34.177819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:34.210382 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f3d3f7-d53c-4a42-8456-438f6faee7b4 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:34.243255 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b336b7c8-3d2c-457a-a8ae-17429aa9f7df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:34.243906 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b336b7c8-3d2c-457a-a8ae-17429aa9f7df None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:34.244184 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b336b7c8-3d2c-457a-a8ae-17429aa9f7df None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:34.244455 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b336b7c8-3d2c-457a-a8ae-17429aa9f7df None admin] PATH_INFO: `/v3/projects/0a8f0912a8064a71bffbd1911c13e3bb` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:34.245562 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b336b7c8-3d2c-457a-a8ae-17429aa9f7df None admin] RBAC: Authorizing `identity:get_project(project_id=0a8f0912a8064a71bffbd1911c13e3bb)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:34.247012 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b336b7c8-3d2c-457a-a8ae-17429aa9f7df None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:34.248628 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 587/1184] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:45:34 2022] GET /identity/v3/projects/0a8f0912a8064a71bffbd1911c13e3bb => generated 361 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:34.254774 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe51990-9f1c-441f-bbd5-1e814b25fda0 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:34.354677 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe51990-9f1c-441f-bbd5-1e814b25fda0 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:34.387862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c81dca-80b6-4b2a-abd0-57b0ce4d5610 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:34.388471 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c81dca-80b6-4b2a-abd0-57b0ce4d5610 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:34.388773 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c81dca-80b6-4b2a-abd0-57b0ce4d5610 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:34.389042 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c81dca-80b6-4b2a-abd0-57b0ce4d5610 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:34.389772 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c81dca-80b6-4b2a-abd0-57b0ce4d5610 None admin] RBAC: Adding query filter params (role.id=1e04c9fea65244d28e1e59fc4c64f32f, scope.project.id=0a8f0912a8064a71bffbd1911c13e3bb, user.id=bb13fb11841c4ebe92382b2834548686) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:45:34.390077 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c81dca-80b6-4b2a-abd0-57b0ce4d5610 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:34.391572 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c81dca-80b6-4b2a-abd0-57b0ce4d5610 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:34.397233 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 598/1185] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:45:34 2022] GET /identity/v3/role_assignments?user.id=bb13fb11841c4ebe92382b2834548686&scope.project.id=0a8f0912a8064a71bffbd1911c13e3bb&role.id=1e04c9fea65244d28e1e59fc4c64f32f => generated 259 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:36.542331 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e185d8c4-003f-4809-90c7-3083616ced53 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:36.542925 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e185d8c4-003f-4809-90c7-3083616ced53 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:36.542925 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e185d8c4-003f-4809-90c7-3083616ced53 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:36.544233 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 588/1186] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:45:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:45:36.549747 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6f14b2-9eff-41d0-897a-507694b036d1 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:36.550203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6f14b2-9eff-41d0-897a-507694b036d1 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:36.550301 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6f14b2-9eff-41d0-897a-507694b036d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:36.568366 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-cc6f14b2-9eff-41d0-897a-507694b036d1 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:45:36.569537 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-cc6f14b2-9eff-41d0-897a-507694b036d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:45:36.594644 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:36.637183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 599/1187] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:45:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:45:36.908007 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e185d8c4-003f-4809-90c7-3083616ced53 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:37.010697 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e185d8c4-003f-4809-90c7-3083616ced53 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:37.043499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2400f5-c5f4-44b4-9347-d77242057432 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:37.044128 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2400f5-c5f4-44b4-9347-d77242057432 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:37.044405 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2400f5-c5f4-44b4-9347-d77242057432 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:37.044675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2400f5-c5f4-44b4-9347-d77242057432 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:37.047277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:37.055812 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2400f5-c5f4-44b4-9347-d77242057432 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:45:37.061178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-9b2400f5-c5f4-44b4-9347-d77242057432 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:45:37.061178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:45:37.061178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:45:37.061178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:45:37.061178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:45:37.061178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:45:37.061178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:45:37.061178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:45:37.061178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:45:37.061178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:45:37.061178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:45:37.061178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:45:37.061178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:45:37.061178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:45:37.061178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:45:37.061178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:45:37.062346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:45:37.062346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:45:37.062346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:45:37.062346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:45:37.062346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:45:37.062346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:45:37.062346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:45:37.062346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:45:37.062346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:45:37.062346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:45:37.062346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:45:37.062346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:45:37.062346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:45:37.062346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:45:37.062346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:45:37.062346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:45:37.063631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:45:37.063631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:45:37.063631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:45:37.063631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:45:37.063631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:45:37.063631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:45:37.063631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:45:37.063631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:45:37.063631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:45:37.063631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:45:37.063631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:45:37.063631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:45:37.063631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 589/1188] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:45:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:45:37.068404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6f14b2-9eff-41d0-897a-507694b036d1 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:37.165728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6f14b2-9eff-41d0-897a-507694b036d1 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:37.199392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd640240-11c8-4228-ab86-b04722f5a35b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:37.199992 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd640240-11c8-4228-ab86-b04722f5a35b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:37.200267 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd640240-11c8-4228-ab86-b04722f5a35b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:37.200534 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd640240-11c8-4228-ab86-b04722f5a35b None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:37.201184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd640240-11c8-4228-ab86-b04722f5a35b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:45:37.201477 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd640240-11c8-4228-ab86-b04722f5a35b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:37.202866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd640240-11c8-4228-ab86-b04722f5a35b None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:37.210630 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 600/1189] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:45:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:37.217731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2400f5-c5f4-44b4-9347-d77242057432 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:37.314248 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2400f5-c5f4-44b4-9347-d77242057432 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:37.349720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5a89c0-06c9-4ed2-a775-cee799e23597 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:37.350347 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5a89c0-06c9-4ed2-a775-cee799e23597 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:37.350656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5a89c0-06c9-4ed2-a775-cee799e23597 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:37.350933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5a89c0-06c9-4ed2-a775-cee799e23597 None admin] PATH_INFO: `/v3/users/bb13fb11841c4ebe92382b2834548686` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:37.352080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5a89c0-06c9-4ed2-a775-cee799e23597 None admin] RBAC: Authorizing `identity:get_user(user_id=bb13fb11841c4ebe92382b2834548686)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:37.353509 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5a89c0-06c9-4ed2-a775-cee799e23597 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:37.355136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 590/1190] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:45:37 2022] GET /identity/v3/users/bb13fb11841c4ebe92382b2834548686 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:37.361152 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd640240-11c8-4228-ab86-b04722f5a35b None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:37.386329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:37.468001 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd640240-11c8-4228-ab86-b04722f5a35b None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:37.501069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe999b1-fdba-4178-bc01-102c5874736a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:37.501694 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe999b1-fdba-4178-bc01-102c5874736a None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:37.501972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe999b1-fdba-4178-bc01-102c5874736a None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:37.502243 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe999b1-fdba-4178-bc01-102c5874736a None admin] PATH_INFO: `/v3/projects/0a8f0912a8064a71bffbd1911c13e3bb` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:37.503442 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe999b1-fdba-4178-bc01-102c5874736a None admin] RBAC: Authorizing `identity:get_project(project_id=0a8f0912a8064a71bffbd1911c13e3bb)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:37.504860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe999b1-fdba-4178-bc01-102c5874736a None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:37.506500 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 601/1191] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:45:37 2022] GET /identity/v3/projects/0a8f0912a8064a71bffbd1911c13e3bb => generated 361 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:37.512523 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5a89c0-06c9-4ed2-a775-cee799e23597 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:37.573217 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:37.611197 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5a89c0-06c9-4ed2-a775-cee799e23597 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:37.643866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8ff229-3968-4d9c-9ed6-7631ea04eb7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:37.644522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8ff229-3968-4d9c-9ed6-7631ea04eb7d None admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:37.644800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8ff229-3968-4d9c-9ed6-7631ea04eb7d None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:37.645069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8ff229-3968-4d9c-9ed6-7631ea04eb7d None admin] PATH_INFO: `/v3/projects/0a8f0912a8064a71bffbd1911c13e3bb/users/bb13fb11841c4ebe92382b2834548686/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:37.646981 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8ff229-3968-4d9c-9ed6-7631ea04eb7d None admin] RBAC: Authorizing `identity:create_grant(project_id=0a8f0912a8064a71bffbd1911c13e3bb, user_id=bb13fb11841c4ebe92382b2834548686, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:37.648402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8ff229-3968-4d9c-9ed6-7631ea04eb7d None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:37.661578 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 591/1192] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:45:37 2022] PUT /identity/v3/projects/0a8f0912a8064a71bffbd1911c13e3bb/users/bb13fb11841c4ebe92382b2834548686/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:45:40.010900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89ce438-7068-457e-bf31-aab82e69676f None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:40.011506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89ce438-7068-457e-bf31-aab82e69676f None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:40.011506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89ce438-7068-457e-bf31-aab82e69676f None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:40.012991 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 602/1193] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:45:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:45:40.019161 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900dd8fa-e9fb-4438-bc87-b6f87518fcef None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:40.019807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900dd8fa-e9fb-4438-bc87-b6f87518fcef None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:40.019807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900dd8fa-e9fb-4438-bc87-b6f87518fcef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:40.039456 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-900dd8fa-e9fb-4438-bc87-b6f87518fcef None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:45:40.040748 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-900dd8fa-e9fb-4438-bc87-b6f87518fcef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:45:40.116120 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 592/1194] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:45:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:45:40.426269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89ce438-7068-457e-bf31-aab82e69676f None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:40.532959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:40.544497 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89ce438-7068-457e-bf31-aab82e69676f None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:40.582060 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2745b5-1e3c-41b2-966e-b2c9a4c3811c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:40.582711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2745b5-1e3c-41b2-966e-b2c9a4c3811c None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:40.582992 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2745b5-1e3c-41b2-966e-b2c9a4c3811c None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:40.583264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2745b5-1e3c-41b2-966e-b2c9a4c3811c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:40.594957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2745b5-1e3c-41b2-966e-b2c9a4c3811c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:45:40.600365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-1f2745b5-1e3c-41b2-966e-b2c9a4c3811c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:45:40.600365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:45:40.600365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:45:40.600365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:45:40.600365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:45:40.600365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:45:40.600365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:45:40.600365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:45:40.600365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:45:40.600365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:45:40.600365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:45:40.600365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:45:40.600365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:45:40.600365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:45:40.600365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:45:40.600365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:45:40.601922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:45:40.601922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:45:40.601922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:45:40.601922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:45:40.601922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:45:40.601922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:45:40.601922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:45:40.601922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:45:40.601922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:45:40.601922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:45:40.601922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:45:40.601922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:45:40.601922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:45:40.601922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:45:40.601922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:45:40.601922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:45:40.603831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:45:40.603831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:45:40.603831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:45:40.603831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:45:40.603831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:45:40.603831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:45:40.603831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:45:40.603831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:45:40.603831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:45:40.603831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:45:40.603831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:45:40.603831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:45:40.603831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 603/1195] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:45:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:45:40.609413 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900dd8fa-e9fb-4438-bc87-b6f87518fcef None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:40.669362 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:40.729992 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900dd8fa-e9fb-4438-bc87-b6f87518fcef None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:40.773424 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e403353d-b938-4b73-8b3c-97ddd34c34bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:40.774112 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e403353d-b938-4b73-8b3c-97ddd34c34bd None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:40.774418 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e403353d-b938-4b73-8b3c-97ddd34c34bd None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:40.774780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e403353d-b938-4b73-8b3c-97ddd34c34bd None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:40.775562 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e403353d-b938-4b73-8b3c-97ddd34c34bd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:45:40.775867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e403353d-b938-4b73-8b3c-97ddd34c34bd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:40.777341 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e403353d-b938-4b73-8b3c-97ddd34c34bd None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:40.787367 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 593/1196] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:45:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:40.793966 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2745b5-1e3c-41b2-966e-b2c9a4c3811c None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:40.941538 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2745b5-1e3c-41b2-966e-b2c9a4c3811c None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:40.979135 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5061e9d4-d635-4f74-8a46-1f05b2923fe4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:40.980059 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5061e9d4-d635-4f74-8a46-1f05b2923fe4 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:40.980405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5061e9d4-d635-4f74-8a46-1f05b2923fe4 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:40.980682 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5061e9d4-d635-4f74-8a46-1f05b2923fe4 None admin] PATH_INFO: `/v3/users/bb13fb11841c4ebe92382b2834548686` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:40.982019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5061e9d4-d635-4f74-8a46-1f05b2923fe4 None admin] RBAC: Authorizing `identity:get_user(user_id=bb13fb11841c4ebe92382b2834548686)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:40.983781 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5061e9d4-d635-4f74-8a46-1f05b2923fe4 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:40.985709 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 604/1197] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 18 00:45:40 2022] GET /identity/v3/users/bb13fb11841c4ebe92382b2834548686 => generated 311 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:40.992210 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e403353d-b938-4b73-8b3c-97ddd34c34bd None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:41.096111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e403353d-b938-4b73-8b3c-97ddd34c34bd None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:41.112149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:41.133421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99ca6f5-c8a0-44b6-b7fe-4e160c1e5c96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:41.134287 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99ca6f5-c8a0-44b6-b7fe-4e160c1e5c96 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:41.135150 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99ca6f5-c8a0-44b6-b7fe-4e160c1e5c96 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:41.135423 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99ca6f5-c8a0-44b6-b7fe-4e160c1e5c96 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:41.236070 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c99ca6f5-c8a0-44b6-b7fe-4e160c1e5c96 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:41.237628 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c99ca6f5-c8a0-44b6-b7fe-4e160c1e5c96 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:41.362102 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 594/1198] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:45:40 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 18 00:45:41.368745 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5061e9d4-d635-4f74-8a46-1f05b2923fe4 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:41.393820 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:41.476791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5061e9d4-d635-4f74-8a46-1f05b2923fe4 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:41.529094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f51ca24-e6b6-4ad8-87f1-1b957b75da78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:41.529752 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f51ca24-e6b6-4ad8-87f1-1b957b75da78 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:41.530033 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f51ca24-e6b6-4ad8-87f1-1b957b75da78 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:41.530308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f51ca24-e6b6-4ad8-87f1-1b957b75da78 None admin] PATH_INFO: `/v3/projects/0a8f0912a8064a71bffbd1911c13e3bb` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:41.531557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f51ca24-e6b6-4ad8-87f1-1b957b75da78 None admin] RBAC: Authorizing `identity:get_project(project_id=0a8f0912a8064a71bffbd1911c13e3bb)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:41.533048 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f51ca24-e6b6-4ad8-87f1-1b957b75da78 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:41.534844 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 605/1199] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 18 00:45:41 2022] GET /identity/v3/projects/0a8f0912a8064a71bffbd1911c13e3bb => generated 361 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:41.541985 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99ca6f5-c8a0-44b6-b7fe-4e160c1e5c96 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:45:41.562877 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:41.652843 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99ca6f5-c8a0-44b6-b7fe-4e160c1e5c96 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:45:41.688758 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f31542-fe68-4242-975e-3d1f981e3e0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:45:41.689364 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f31542-fe68-4242-975e-3d1f981e3e0a None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:45:41.689639 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f31542-fe68-4242-975e-3d1f981e3e0a None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:45:41.689910 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f31542-fe68-4242-975e-3d1f981e3e0a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:45:41.690674 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f31542-fe68-4242-975e-3d1f981e3e0a None admin] RBAC: Adding query filter params (role.id=1e04c9fea65244d28e1e59fc4c64f32f, scope.project.id=0a8f0912a8064a71bffbd1911c13e3bb, user.id=bb13fb11841c4ebe92382b2834548686) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:45:41.690984 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f31542-fe68-4242-975e-3d1f981e3e0a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:45:41.692454 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f31542-fe68-4242-975e-3d1f981e3e0a None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:45:41.698989 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 595/1200] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:45:41 2022] GET /identity/v3/role_assignments?user.id=bb13fb11841c4ebe92382b2834548686&scope.project.id=0a8f0912a8064a71bffbd1911c13e3bb&role.id=1e04c9fea65244d28e1e59fc4c64f32f => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:45:51.524370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=56 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:45:51.695960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=60 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:40.043842 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab90de9f-0319-4584-a9c7-0c3a75171647 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:40.044780 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab90de9f-0319-4584-a9c7-0c3a75171647 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:40.044780 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab90de9f-0319-4584-a9c7-0c3a75171647 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:40.047215 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 606/1201] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:46:40 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:46:40.054631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85182332-7a3f-4589-8aeb-a79ad84e97f0 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:40.055730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85182332-7a3f-4589-8aeb-a79ad84e97f0 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:40.055730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85182332-7a3f-4589-8aeb-a79ad84e97f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:40.081202 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-85182332-7a3f-4589-8aeb-a79ad84e97f0 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:46:40.082816 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-85182332-7a3f-4589-8aeb-a79ad84e97f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:46:40.215921 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 596/1202] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:46:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:46:40.534822 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab90de9f-0319-4584-a9c7-0c3a75171647 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:40.741349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab90de9f-0319-4584-a9c7-0c3a75171647 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:40.822392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4d9ff7-7d39-4957-9b09-c4827000472a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:40.823810 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4d9ff7-7d39-4957-9b09-c4827000472a None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:40.824294 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4d9ff7-7d39-4957-9b09-c4827000472a None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:40.824826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4d9ff7-7d39-4957-9b09-c4827000472a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:40.843664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4d9ff7-7d39-4957-9b09-c4827000472a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:40.843664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4d9ff7-7d39-4957-9b09-c4827000472a None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:40.867057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-3a4d9ff7-7d39-4957-9b09-c4827000472a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:46:40.867057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:46:40.867057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:46:40.867057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:46:40.867057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:46:40.867057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:46:40.867057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:46:40.867057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:46:40.867057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:46:40.867057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:46:40.867057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:46:40.867057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:46:40.867057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:46:40.867057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:46:40.867057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:46:40.867057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:46:40.875887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:46:40.875887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:46:40.875887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:46:40.875887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:46:40.875887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:46:40.875887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:46:40.875887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:46:40.875887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:46:40.875887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:46:40.875887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:46:40.875887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:46:40.875887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:46:40.875887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:46:40.875887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:46:40.875887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:46:40.875887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:46:40.875887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:46:40.877744 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:46:40.877744 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:46:40.877744 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:46:40.877744 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 607/1203] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:46:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 335 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:46:40.882790 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85182332-7a3f-4589-8aeb-a79ad84e97f0 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:41.107257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85182332-7a3f-4589-8aeb-a79ad84e97f0 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:41.174282 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:41.193587 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c911e66-39e8-4732-ade5-3d307e21abdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:41.194688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c911e66-39e8-4732-ade5-3d307e21abdd None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:41.195228 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c911e66-39e8-4732-ade5-3d307e21abdd None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:41.195661 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c911e66-39e8-4732-ade5-3d307e21abdd None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:41.196949 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c911e66-39e8-4732-ade5-3d307e21abdd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:46:41.197692 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c911e66-39e8-4732-ade5-3d307e21abdd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:41.199794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c911e66-39e8-4732-ade5-3d307e21abdd None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:41.223396 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 597/1204] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:46:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 344 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:46:41.233085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4d9ff7-7d39-4957-9b09-c4827000472a None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:41.338222 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:41.434979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4d9ff7-7d39-4957-9b09-c4827000472a None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:41.506182 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcca393b-26de-4a70-b4ef-34727ccd2e0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:41.506941 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcca393b-26de-4a70-b4ef-34727ccd2e0e None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:41.507188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcca393b-26de-4a70-b4ef-34727ccd2e0e None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:41.507492 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcca393b-26de-4a70-b4ef-34727ccd2e0e None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:41.513343 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcca393b-26de-4a70-b4ef-34727ccd2e0e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:41.513343 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcca393b-26de-4a70-b4ef-34727ccd2e0e None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:41.543230 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 608/1205] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 18 00:46:41 2022] POST /identity/v3/users => generated 251 bytes in 311 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:46:44.305087 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea45f9d-0166-4f1e-b5c8-25d99852262d None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:44.307565 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea45f9d-0166-4f1e-b5c8-25d99852262d None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:44.307565 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea45f9d-0166-4f1e-b5c8-25d99852262d None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:44.308235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 598/1206] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:46:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:46:44.315596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fef986-649b-40cd-9405-7b3e37a968cb None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:44.322358 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fef986-649b-40cd-9405-7b3e37a968cb None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:44.322358 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fef986-649b-40cd-9405-7b3e37a968cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:44.344221 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-11fef986-649b-40cd-9405-7b3e37a968cb None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:46:44.345889 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-11fef986-649b-40cd-9405-7b3e37a968cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:46:44.426510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 609/1207] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:46:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:46:44.730942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea45f9d-0166-4f1e-b5c8-25d99852262d None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:44.869932 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea45f9d-0166-4f1e-b5c8-25d99852262d None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:44.924865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be213a2-7884-44e9-9bbd-f9053f7b5633 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:44.925657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be213a2-7884-44e9-9bbd-f9053f7b5633 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:44.926996 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be213a2-7884-44e9-9bbd-f9053f7b5633 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:44.927129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be213a2-7884-44e9-9bbd-f9053f7b5633 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:44.947401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be213a2-7884-44e9-9bbd-f9053f7b5633 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:46:44.952519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-1be213a2-7884-44e9-9bbd-f9053f7b5633 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:46:44.952519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:46:44.952519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:46:44.952519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:46:44.952519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:46:44.952519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:46:44.952519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:46:44.952519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:46:44.952519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:46:44.952519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:46:44.952519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:46:44.952519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:46:44.952519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:46:44.952519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:46:44.952519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:46:44.952519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:46:44.953971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:46:44.953971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:46:44.953971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:46:44.953971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:46:44.953971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:46:44.953971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:46:44.953971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:46:44.953971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:46:44.953971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:46:44.953971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:46:44.953971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:46:44.953971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:46:44.953971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:46:44.953971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:46:44.953971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:46:44.953971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:46:44.955125 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:46:44.955125 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:46:44.955125 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:46:44.955125 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:46:44.955125 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:46:44.955125 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:46:44.955125 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:46:44.955125 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:46:44.955125 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:46:44.955125 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:46:44.955125 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:46:44.955125 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:46:44.955125 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 599/1208] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:46:44 2022] GET /identity/v3/roles/service => generated 85 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:46:44.962120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fef986-649b-40cd-9405-7b3e37a968cb None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:45.026749 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:45.128504 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fef986-649b-40cd-9405-7b3e37a968cb None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:45.178888 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dd01ea-6e2d-44f0-be40-53745728a9c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:45.179670 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dd01ea-6e2d-44f0-be40-53745728a9c8 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:45.180012 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dd01ea-6e2d-44f0-be40-53745728a9c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:45.180270 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dd01ea-6e2d-44f0-be40-53745728a9c8 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:45.181292 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24dd01ea-6e2d-44f0-be40-53745728a9c8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:46:45.181601 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24dd01ea-6e2d-44f0-be40-53745728a9c8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:45.183267 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24dd01ea-6e2d-44f0-be40-53745728a9c8 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:45.194527 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 610/1209] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 18 00:46:44 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:46:45.201768 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be213a2-7884-44e9-9bbd-f9053f7b5633 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:45.279315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:45.376368 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be213a2-7884-44e9-9bbd-f9053f7b5633 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:45.432889 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bc4d32-d9a9-4046-847f-9124ac4c0d6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:45.433836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bc4d32-d9a9-4046-847f-9124ac4c0d6e None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:45.434173 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bc4d32-d9a9-4046-847f-9124ac4c0d6e None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:45.434632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bc4d32-d9a9-4046-847f-9124ac4c0d6e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:45.448369 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19bc4d32-d9a9-4046-847f-9124ac4c0d6e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:45.450123 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19bc4d32-d9a9-4046-847f-9124ac4c0d6e None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:45.479414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-19bc4d32-d9a9-4046-847f-9124ac4c0d6e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:46:45.479414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:46:45.479414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:46:45.479414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:46:45.479414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:46:45.479414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:46:45.479414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:46:45.479414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:46:45.479414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:46:45.479414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:46:45.479414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:46:45.479414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:46:45.479414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:46:45.479414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:46:45.479414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:46:45.479414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:46:45.480670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:46:45.480670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:46:45.480670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:46:45.480670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:46:45.480670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:46:45.480670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:46:45.480670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:46:45.480670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:46:45.480670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:46:45.480670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:46:45.480670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:46:45.480670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:46:45.480670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:46:45.480670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:46:45.480670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:46:45.480670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:46:45.480670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:46:45.481711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:46:45.481711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:46:45.481711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:46:45.481711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 600/1210] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:46:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:46:45.487488 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dd01ea-6e2d-44f0-be40-53745728a9c8 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:45.631816 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:45.635658 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dd01ea-6e2d-44f0-be40-53745728a9c8 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:45.679202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b115fb0b-0e40-4d62-9713-d806e24dac76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:45.680144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b115fb0b-0e40-4d62-9713-d806e24dac76 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:45.681404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b115fb0b-0e40-4d62-9713-d806e24dac76 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:45.681878 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b115fb0b-0e40-4d62-9713-d806e24dac76 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:45.682754 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b115fb0b-0e40-4d62-9713-d806e24dac76 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:46:45.683213 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b115fb0b-0e40-4d62-9713-d806e24dac76 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:45.684810 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b115fb0b-0e40-4d62-9713-d806e24dac76 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:45.699155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 611/1211] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:46:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:46:45.705399 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bc4d32-d9a9-4046-847f-9124ac4c0d6e None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:45.872885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bc4d32-d9a9-4046-847f-9124ac4c0d6e None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:45.874851 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:45.919490 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a0d2a8-1fad-4df4-bba8-d919b647160f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:45.919490 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a0d2a8-1fad-4df4-bba8-d919b647160f None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:45.919490 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a0d2a8-1fad-4df4-bba8-d919b647160f None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:45.919490 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a0d2a8-1fad-4df4-bba8-d919b647160f None admin] PATH_INFO: `/v3/users/placement` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:45.925222 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a0d2a8-1fad-4df4-bba8-d919b647160f None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:45.926854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a0d2a8-1fad-4df4-bba8-d919b647160f None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:45.941135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-a8a0d2a8-1fad-4df4-bba8-d919b647160f None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 18 00:46:45.941135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:46:45.941135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:46:45.941135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:46:45.941135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:46:45.941135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:46:45.941135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:46:45.941135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:46:45.941135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:46:45.941135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:46:45.941135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:46:45.941135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:46:45.941135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:46:45.941135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:46:45.941135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:46:45.941135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:46:45.942850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:46:45.942850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:46:45.942850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:46:45.942850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:46:45.942850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:46:45.942850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:46:45.942850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:46:45.942850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:46:45.942850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:46:45.942850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:46:45.942850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:46:45.942850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:46:45.942850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:46:45.942850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:46:45.942850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:46:45.942850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:46:45.942850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:46:45.942850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:46:45.943958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:46:45.943958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:46:45.943958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:46:45.943958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:46:45.943958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:46:45.943958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:46:45.943958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:46:45.943958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:46:45.943958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 18 00:46:45.943958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:46:45.943958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 601/1212] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:46:45 2022] GET /identity/v3/users/placement => generated 87 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:46:45.950432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b115fb0b-0e40-4d62-9713-d806e24dac76 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:46.091903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b115fb0b-0e40-4d62-9713-d806e24dac76 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:46.137876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748f47bb-a2b1-4c35-b03e-2605318c1926 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:46.138820 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748f47bb-a2b1-4c35-b03e-2605318c1926 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:46.139269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748f47bb-a2b1-4c35-b03e-2605318c1926 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:46.139759 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748f47bb-a2b1-4c35-b03e-2605318c1926 None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:46.140695 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748f47bb-a2b1-4c35-b03e-2605318c1926 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:46:46.141204 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748f47bb-a2b1-4c35-b03e-2605318c1926 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:46.142852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748f47bb-a2b1-4c35-b03e-2605318c1926 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:46.162077 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 612/1213] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Sep 18 00:46:45 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:46:46.170219 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a0d2a8-1fad-4df4-bba8-d919b647160f None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:46.310801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a0d2a8-1fad-4df4-bba8-d919b647160f None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:46.352910 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba87d6d7-d2d3-4ac7-8103-8916a1198b7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:46.353570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba87d6d7-d2d3-4ac7-8103-8916a1198b7a None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:46.353852 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba87d6d7-d2d3-4ac7-8103-8916a1198b7a None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:46.354124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba87d6d7-d2d3-4ac7-8103-8916a1198b7a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:46.391394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:46.443819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba87d6d7-d2d3-4ac7-8103-8916a1198b7a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:46.445600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba87d6d7-d2d3-4ac7-8103-8916a1198b7a None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:46.585857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 602/1214] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:46:46 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 18 00:46:46.594648 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748f47bb-a2b1-4c35-b03e-2605318c1926 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:46.623959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:46.727464 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748f47bb-a2b1-4c35-b03e-2605318c1926 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:46.767888 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e36682-0720-4814-a798-c7023b3750ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:46.768637 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e36682-0720-4814-a798-c7023b3750ad None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:46.768927 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e36682-0720-4814-a798-c7023b3750ad None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:46.769206 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e36682-0720-4814-a798-c7023b3750ad None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:46.785887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93e36682-0720-4814-a798-c7023b3750ad None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:46.789077 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93e36682-0720-4814-a798-c7023b3750ad None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:46.810135 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-93e36682-0720-4814-a798-c7023b3750ad None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:46:46.810135 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:46:46.810135 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:46:46.810135 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:46:46.810135 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:46:46.810135 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:46:46.810135 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:46:46.810135 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:46:46.810135 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:46:46.810135 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:46:46.810135 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:46:46.810135 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:46:46.810135 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:46:46.810135 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:46:46.810135 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:46:46.810135 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:46:46.811595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:46:46.811595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:46:46.811595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:46:46.811595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:46:46.811595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:46:46.811595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:46:46.811595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:46:46.811595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:46:46.811595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:46:46.811595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:46:46.811595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:46:46.811595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:46:46.811595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:46:46.811595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:46:46.811595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:46:46.811595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:46:46.811595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:46:46.812796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:46:46.812796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:46:46.812796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:46:46.812796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 613/1215] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:46:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:46:46.817652 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba87d6d7-d2d3-4ac7-8103-8916a1198b7a None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:46.921308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:46.997033 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba87d6d7-d2d3-4ac7-8103-8916a1198b7a None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:47.038666 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099ca6ed-4b61-4555-bd86-93a89e6743a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:47.039279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099ca6ed-4b61-4555-bd86-93a89e6743a3 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:47.039570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099ca6ed-4b61-4555-bd86-93a89e6743a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:47.039831 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099ca6ed-4b61-4555-bd86-93a89e6743a3 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:47.041069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099ca6ed-4b61-4555-bd86-93a89e6743a3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:46:47.041371 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099ca6ed-4b61-4555-bd86-93a89e6743a3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:47.043032 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099ca6ed-4b61-4555-bd86-93a89e6743a3 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:47.057143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 603/1216] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:46:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:46:47.063444 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e36682-0720-4814-a798-c7023b3750ad None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:47.163752 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:47.201405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e36682-0720-4814-a798-c7023b3750ad None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:47.242228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9498b4b-beaf-42c2-96be-631af152239b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:47.243100 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9498b4b-beaf-42c2-96be-631af152239b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:47.243926 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9498b4b-beaf-42c2-96be-631af152239b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:47.244205 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9498b4b-beaf-42c2-96be-631af152239b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:47.252218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9498b4b-beaf-42c2-96be-631af152239b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:47.253658 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9498b4b-beaf-42c2-96be-631af152239b None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:47.284170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-b9498b4b-beaf-42c2-96be-631af152239b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:46:47.284170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:46:47.284170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:46:47.284170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:46:47.284170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:46:47.284170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:46:47.284170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:46:47.284170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:46:47.284170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:46:47.284170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:46:47.284170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:46:47.284170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:46:47.284170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:46:47.284170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:46:47.284170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:46:47.285434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:46:47.285434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:46:47.285434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:46:47.285434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:46:47.285434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:46:47.285434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:46:47.285434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:46:47.285434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:46:47.285434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:46:47.285434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:46:47.285434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:46:47.285434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:46:47.285434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:46:47.285434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:46:47.285434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:46:47.285434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:46:47.285434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:46:47.286666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:46:47.286666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:46:47.286666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:46:47.286666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:46:47.286666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:46:47.286666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:46:47.286666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:46:47.286666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:46:47.286666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 614/1217] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:46:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:46:47.293316 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099ca6ed-4b61-4555-bd86-93a89e6743a3 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:47.426531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099ca6ed-4b61-4555-bd86-93a89e6743a3 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:47.459385 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:47.472207 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bd314b-73ee-4d5c-9a82-59c15932a663 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:47.472886 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bd314b-73ee-4d5c-9a82-59c15932a663 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:47.473211 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bd314b-73ee-4d5c-9a82-59c15932a663 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:47.473462 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bd314b-73ee-4d5c-9a82-59c15932a663 None admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:47.474272 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bd314b-73ee-4d5c-9a82-59c15932a663 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:46:47.474514 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bd314b-73ee-4d5c-9a82-59c15932a663 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:47.476218 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bd314b-73ee-4d5c-9a82-59c15932a663 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:47.492393 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 604/1218] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:46:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:46:47.500290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9498b4b-beaf-42c2-96be-631af152239b None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:47.667530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9498b4b-beaf-42c2-96be-631af152239b None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:47.708628 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:47.716413 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e0e996-8f3f-48cf-ad3f-662f352c033d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:47.717099 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e0e996-8f3f-48cf-ad3f-662f352c033d None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:47.717308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e0e996-8f3f-48cf-ad3f-662f352c033d None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:47.717579 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e0e996-8f3f-48cf-ad3f-662f352c033d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:47.718311 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e0e996-8f3f-48cf-ad3f-662f352c033d None admin] RBAC: Adding query filter params (role.id=81523e544c7e4993af4d1e06ce3754bc, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=8f30116fb5ce43dab69efa8bafd9d049) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:46:47.718651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e0e996-8f3f-48cf-ad3f-662f352c033d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:47.720093 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e0e996-8f3f-48cf-ad3f-662f352c033d None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:47.726206 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 615/1219] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:46:47 2022] GET /identity/v3/role_assignments?user.id=8f30116fb5ce43dab69efa8bafd9d049&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=81523e544c7e4993af4d1e06ce3754bc => generated 259 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:46:50.202618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2256d38-eb89-4a63-8967-a85ed0901b0e None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:50.203333 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2256d38-eb89-4a63-8967-a85ed0901b0e None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:50.203333 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2256d38-eb89-4a63-8967-a85ed0901b0e None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:50.205146 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 605/1220] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:46:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:46:50.211661 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc257ca3-5968-4b0b-a23a-36723297fede None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:50.212622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc257ca3-5968-4b0b-a23a-36723297fede None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:50.212966 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc257ca3-5968-4b0b-a23a-36723297fede None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:50.239137 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-cc257ca3-5968-4b0b-a23a-36723297fede None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:46:50.241308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-cc257ca3-5968-4b0b-a23a-36723297fede None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:46:50.346986 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 616/1221] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:46:50 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:46:50.660118 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2256d38-eb89-4a63-8967-a85ed0901b0e None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:50.810468 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2256d38-eb89-4a63-8967-a85ed0901b0e None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:50.854487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed0d73c-2e3f-477e-a22d-18f9ceadc189 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:50.855163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed0d73c-2e3f-477e-a22d-18f9ceadc189 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:50.855477 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed0d73c-2e3f-477e-a22d-18f9ceadc189 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:50.855689 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed0d73c-2e3f-477e-a22d-18f9ceadc189 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:50.867178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed0d73c-2e3f-477e-a22d-18f9ceadc189 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:46:50.872397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-0ed0d73c-2e3f-477e-a22d-18f9ceadc189 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:46:50.872397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:46:50.872397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:46:50.872397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:46:50.872397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:46:50.872397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:46:50.872397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:46:50.872397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:46:50.872397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:46:50.872397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:46:50.872397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:46:50.872397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:46:50.872397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:46:50.872397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:46:50.872397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:46:50.872397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:46:50.873807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:46:50.873807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:46:50.873807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:46:50.873807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:46:50.873807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:46:50.873807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:46:50.873807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:46:50.873807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:46:50.873807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:46:50.873807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:46:50.873807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:46:50.873807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:46:50.873807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:46:50.873807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:46:50.873807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:46:50.873807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:46:50.875315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:46:50.875315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:46:50.875315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:46:50.875315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:46:50.875315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:46:50.875315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:46:50.875315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:46:50.875315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:46:50.875315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:46:50.875315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:46:50.875315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:46:50.875315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:46:50.875315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 606/1222] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:46:50 2022] GET /identity/v3/roles/service => generated 85 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:46:50.884310 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc257ca3-5968-4b0b-a23a-36723297fede None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:51.027339 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc257ca3-5968-4b0b-a23a-36723297fede None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:51.061218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:51.074905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1194c286-e673-4644-9a43-bd5233dcedde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:51.075526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1194c286-e673-4644-9a43-bd5233dcedde None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:51.075906 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1194c286-e673-4644-9a43-bd5233dcedde None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:51.076100 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1194c286-e673-4644-9a43-bd5233dcedde None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:51.077617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1194c286-e673-4644-9a43-bd5233dcedde None admin] RBAC: Adding query filter params (name=service) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:46:51.077617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1194c286-e673-4644-9a43-bd5233dcedde None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:51.078640 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1194c286-e673-4644-9a43-bd5233dcedde None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:51.089309 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 617/1223] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 18 00:46:50 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:46:51.095704 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed0d73c-2e3f-477e-a22d-18f9ceadc189 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:51.166817 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:51.257097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed0d73c-2e3f-477e-a22d-18f9ceadc189 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:51.307644 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14f4b50-668c-4193-b5da-db29511b6f3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:51.308712 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14f4b50-668c-4193-b5da-db29511b6f3a None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:51.309210 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14f4b50-668c-4193-b5da-db29511b6f3a None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:51.309652 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14f4b50-668c-4193-b5da-db29511b6f3a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:51.327855 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14f4b50-668c-4193-b5da-db29511b6f3a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:51.330376 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14f4b50-668c-4193-b5da-db29511b6f3a None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:51.358111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-a14f4b50-668c-4193-b5da-db29511b6f3a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:46:51.358111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:46:51.358111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:46:51.358111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:46:51.358111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:46:51.358111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:46:51.358111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:46:51.358111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:46:51.358111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:46:51.358111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:46:51.358111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:46:51.358111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:46:51.358111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:46:51.358111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:46:51.358111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:46:51.358111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:46:51.363724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:46:51.363724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:46:51.363724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:46:51.363724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:46:51.363724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:46:51.363724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:46:51.363724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:46:51.363724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:46:51.363724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:46:51.363724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:46:51.363724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:46:51.363724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:46:51.363724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:46:51.363724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:46:51.363724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:46:51.363724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:46:51.363724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:46:51.365445 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:46:51.365445 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:46:51.365445 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:46:51.365445 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 607/1224] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:46:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:46:51.366909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1194c286-e673-4644-9a43-bd5233dcedde None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:51.509549 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1194c286-e673-4644-9a43-bd5233dcedde None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:51.564804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3086fc30-3661-4dda-b793-30c7f5a55105 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:51.565595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3086fc30-3661-4dda-b793-30c7f5a55105 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:51.565985 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3086fc30-3661-4dda-b793-30c7f5a55105 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:51.566499 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3086fc30-3661-4dda-b793-30c7f5a55105 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:51.567655 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3086fc30-3661-4dda-b793-30c7f5a55105 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:46:51.568199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3086fc30-3661-4dda-b793-30c7f5a55105 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:51.570206 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3086fc30-3661-4dda-b793-30c7f5a55105 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:51.585924 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 618/1225] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:46:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:46:51.592537 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14f4b50-668c-4193-b5da-db29511b6f3a None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:51.743866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:51.752723 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14f4b50-668c-4193-b5da-db29511b6f3a None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:51.796077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe02cc4-76fc-4da8-a6b1-f6584ae0b7f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:51.797066 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe02cc4-76fc-4da8-a6b1-f6584ae0b7f5 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:51.797517 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe02cc4-76fc-4da8-a6b1-f6584ae0b7f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:51.797897 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe02cc4-76fc-4da8-a6b1-f6584ae0b7f5 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:51.807726 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe02cc4-76fc-4da8-a6b1-f6584ae0b7f5 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:51.809258 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe02cc4-76fc-4da8-a6b1-f6584ae0b7f5 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:51.824498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-6fe02cc4-76fc-4da8-a6b1-f6584ae0b7f5 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 18 00:46:51.824498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:46:51.824498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:46:51.824498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:46:51.824498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:46:51.824498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:46:51.824498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:46:51.824498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:46:51.824498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:46:51.824498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:46:51.824498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:46:51.824498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:46:51.824498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:46:51.824498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:46:51.824498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:46:51.824498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:46:51.825668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:46:51.825668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:46:51.825668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:46:51.825668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:46:51.825668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:46:51.825668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:46:51.825668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:46:51.825668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:46:51.825668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:46:51.825668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:46:51.825668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:46:51.825668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:46:51.825668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:46:51.825668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:46:51.825668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:46:51.825668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:46:51.825668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:46:51.825668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:46:51.826827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:46:51.826827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:46:51.826827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:46:51.826827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:46:51.826827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:46:51.826827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:46:51.826827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:46:51.826827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:46:51.826827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 18 00:46:51.826827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:46:51.830910 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 608/1226] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:46:51 2022] GET /identity/v3/users/placement => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:46:51.831956 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3086fc30-3661-4dda-b793-30c7f5a55105 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:51.901080 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:52.003036 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3086fc30-3661-4dda-b793-30c7f5a55105 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:52.056486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92c7d87-892b-4e5f-a6fc-91704c8149b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:52.057145 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92c7d87-892b-4e5f-a6fc-91704c8149b2 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:52.057554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92c7d87-892b-4e5f-a6fc-91704c8149b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:52.057859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92c7d87-892b-4e5f-a6fc-91704c8149b2 None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:52.058733 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92c7d87-892b-4e5f-a6fc-91704c8149b2 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:46:52.059043 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92c7d87-892b-4e5f-a6fc-91704c8149b2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:52.060618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92c7d87-892b-4e5f-a6fc-91704c8149b2 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:52.086202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 619/1227] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Sep 18 00:46:51 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:46:52.088065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe02cc4-76fc-4da8-a6b1-f6584ae0b7f5 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:52.214542 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe02cc4-76fc-4da8-a6b1-f6584ae0b7f5 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:52.265901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8e9a6b-9a47-4bdf-bf7f-1c9ccddcc34d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:52.266670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8e9a6b-9a47-4bdf-bf7f-1c9ccddcc34d None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:52.266948 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8e9a6b-9a47-4bdf-bf7f-1c9ccddcc34d None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:52.267135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8e9a6b-9a47-4bdf-bf7f-1c9ccddcc34d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:52.286914 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8e9a6b-9a47-4bdf-bf7f-1c9ccddcc34d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:52.288526 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8e9a6b-9a47-4bdf-bf7f-1c9ccddcc34d None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:52.323618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-cd8e9a6b-9a47-4bdf-bf7f-1c9ccddcc34d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:46:52.323618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:46:52.323618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:46:52.323618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:46:52.323618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:46:52.323618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:46:52.323618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:46:52.323618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:46:52.323618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:46:52.323618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:46:52.323618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:46:52.323618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:46:52.323618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:46:52.323618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:46:52.323618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:46:52.323618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:46:52.324864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:46:52.324864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:46:52.324864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:46:52.324864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:46:52.324864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:46:52.324864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:46:52.324864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:46:52.324864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:46:52.324864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:46:52.324864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:46:52.324864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:46:52.324864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:46:52.324864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:46:52.324864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:46:52.324864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:46:52.324864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:46:52.324864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:46:52.326115 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:46:52.326115 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:46:52.326115 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:46:52.334531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 609/1228] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:46:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:46:52.341441 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92c7d87-892b-4e5f-a6fc-91704c8149b2 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:52.482142 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92c7d87-892b-4e5f-a6fc-91704c8149b2 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:52.486881 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:52.550750 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6143d39c-9f29-4216-a3fd-7dda899e3ac4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:52.551625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6143d39c-9f29-4216-a3fd-7dda899e3ac4 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:52.552058 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6143d39c-9f29-4216-a3fd-7dda899e3ac4 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:52.552514 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6143d39c-9f29-4216-a3fd-7dda899e3ac4 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:52.553393 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6143d39c-9f29-4216-a3fd-7dda899e3ac4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:46:52.553817 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6143d39c-9f29-4216-a3fd-7dda899e3ac4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:52.556064 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6143d39c-9f29-4216-a3fd-7dda899e3ac4 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:52.572980 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 620/1229] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:46:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:46:52.581098 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8e9a6b-9a47-4bdf-bf7f-1c9ccddcc34d None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:52.593610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:52.719549 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8e9a6b-9a47-4bdf-bf7f-1c9ccddcc34d None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:52.765617 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e058657-f869-46fc-ad5a-101718640a71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:52.766227 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e058657-f869-46fc-ad5a-101718640a71 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:52.766552 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e058657-f869-46fc-ad5a-101718640a71 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:52.766910 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e058657-f869-46fc-ad5a-101718640a71 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:52.774828 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e058657-f869-46fc-ad5a-101718640a71 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:52.776544 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e058657-f869-46fc-ad5a-101718640a71 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:52.789410 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-9e058657-f869-46fc-ad5a-101718640a71 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:46:52.789410 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:46:52.789410 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:46:52.789410 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:46:52.789410 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:46:52.789410 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:46:52.789410 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:46:52.789410 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:46:52.789410 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:46:52.789410 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:46:52.789410 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:46:52.789410 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:46:52.789410 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:46:52.789410 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:46:52.789410 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:46:52.791736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:46:52.791736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:46:52.791736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:46:52.791736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:46:52.791736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:46:52.791736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:46:52.791736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:46:52.791736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:46:52.791736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:46:52.791736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:46:52.791736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:46:52.791736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:46:52.791736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:46:52.791736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:46:52.791736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:46:52.791736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:46:52.791736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:46:52.793167 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:46:52.793167 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:46:52.793167 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:46:52.793167 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:46:52.793167 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:46:52.793167 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:46:52.793167 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:46:52.793167 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:46:52.793167 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 610/1230] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:46:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:46:52.797328 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6143d39c-9f29-4216-a3fd-7dda899e3ac4 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:52.933824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6143d39c-9f29-4216-a3fd-7dda899e3ac4 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:52.992742 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7789e3-8274-46e3-890f-fc06dc2d32b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:52.994835 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7789e3-8274-46e3-890f-fc06dc2d32b5 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:52.995322 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7789e3-8274-46e3-890f-fc06dc2d32b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:52.995872 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7789e3-8274-46e3-890f-fc06dc2d32b5 None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:52.996896 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7789e3-8274-46e3-890f-fc06dc2d32b5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:46:52.997485 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7789e3-8274-46e3-890f-fc06dc2d32b5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:52.999776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7789e3-8274-46e3-890f-fc06dc2d32b5 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:53.020922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 621/1231] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:46:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:46:53.027838 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e058657-f869-46fc-ad5a-101718640a71 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:53.107706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:53.159488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e058657-f869-46fc-ad5a-101718640a71 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:53.204698 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2175b92-27ac-4fc8-95d4-7d1805e50a20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:53.205353 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2175b92-27ac-4fc8-95d4-7d1805e50a20 None admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:53.205630 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2175b92-27ac-4fc8-95d4-7d1805e50a20 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:53.205904 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2175b92-27ac-4fc8-95d4-7d1805e50a20 None admin] PATH_INFO: `/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/8f30116fb5ce43dab69efa8bafd9d049/roles/81523e544c7e4993af4d1e06ce3754bc` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:53.223956 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2175b92-27ac-4fc8-95d4-7d1805e50a20 None admin] RBAC: Authorizing `identity:create_grant(project_id=c520a8da3dc54adba4f7cc04ef42e367, user_id=8f30116fb5ce43dab69efa8bafd9d049, role_id=81523e544c7e4993af4d1e06ce3754bc)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:53.225413 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2175b92-27ac-4fc8-95d4-7d1805e50a20 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:53.240778 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 611/1232] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:46:53 2022] PUT /identity/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/8f30116fb5ce43dab69efa8bafd9d049/roles/81523e544c7e4993af4d1e06ce3754bc => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:46:55.566545 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5245ebe-52fe-4cfa-a8cf-5f04e9e0e8b7 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:55.567088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5245ebe-52fe-4cfa-a8cf-5f04e9e0e8b7 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:55.567088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5245ebe-52fe-4cfa-a8cf-5f04e9e0e8b7 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:55.568499 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 622/1233] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:46:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:46:55.575123 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5d1f26-e85d-47c9-b4c5-adb889fb708c None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:55.575622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5d1f26-e85d-47c9-b4c5-adb889fb708c None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:55.575714 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5d1f26-e85d-47c9-b4c5-adb889fb708c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:55.596948 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-cc5d1f26-e85d-47c9-b4c5-adb889fb708c None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:46:55.598367 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-cc5d1f26-e85d-47c9-b4c5-adb889fb708c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:46:55.675161 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 612/1234] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:46:55 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 18 00:46:55.976799 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5245ebe-52fe-4cfa-a8cf-5f04e9e0e8b7 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:56.017424 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:56.106801 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5245ebe-52fe-4cfa-a8cf-5f04e9e0e8b7 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:56.148684 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3205431a-b209-4c82-b3f6-4958179b5ab3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:56.149323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3205431a-b209-4c82-b3f6-4958179b5ab3 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:56.149629 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3205431a-b209-4c82-b3f6-4958179b5ab3 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:56.149912 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3205431a-b209-4c82-b3f6-4958179b5ab3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:56.161699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3205431a-b209-4c82-b3f6-4958179b5ab3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:46:56.167202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-3205431a-b209-4c82-b3f6-4958179b5ab3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:46:56.167202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:46:56.167202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:46:56.167202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:46:56.167202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:46:56.167202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:46:56.167202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:46:56.167202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:46:56.167202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:46:56.167202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:46:56.167202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:46:56.167202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:46:56.167202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:46:56.167202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:46:56.167202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:46:56.167202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:46:56.168848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:46:56.168848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:46:56.168848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:46:56.168848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:46:56.168848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:46:56.168848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:46:56.168848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:46:56.168848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:46:56.168848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:46:56.168848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:46:56.168848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:46:56.168848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:46:56.168848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:46:56.168848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:46:56.168848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:46:56.168848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:46:56.170685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:46:56.170685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:46:56.170685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:46:56.170685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:46:56.170685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:46:56.170685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:46:56.170685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:46:56.170685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:46:56.170685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:46:56.170685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:46:56.170685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 18 00:46:56.170685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:46:56.170685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 623/1235] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 18 00:46:55 2022] GET /identity/v3/roles/service => generated 85 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:46:56.175064 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5d1f26-e85d-47c9-b4c5-adb889fb708c None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:56.243044 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:56.316293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5d1f26-e85d-47c9-b4c5-adb889fb708c None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:56.359417 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e986b207-e6e8-4ff9-846e-1159ab2361e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:56.360169 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e986b207-e6e8-4ff9-846e-1159ab2361e7 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:56.360530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e986b207-e6e8-4ff9-846e-1159ab2361e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:56.360837 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e986b207-e6e8-4ff9-846e-1159ab2361e7 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:56.361711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e986b207-e6e8-4ff9-846e-1159ab2361e7 None admin] RBAC: Adding query filter params (name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:46:56.362073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e986b207-e6e8-4ff9-846e-1159ab2361e7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:56.363582 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e986b207-e6e8-4ff9-846e-1159ab2361e7 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:56.372803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 613/1236] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 18 00:46:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:46:56.381872 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3205431a-b209-4c82-b3f6-4958179b5ab3 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:56.484873 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:56.504752 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3205431a-b209-4c82-b3f6-4958179b5ab3 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:56.547295 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d3408d-d912-473f-a2b0-98ea776c48af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:56.548333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d3408d-d912-473f-a2b0-98ea776c48af None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:56.548817 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d3408d-d912-473f-a2b0-98ea776c48af None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:56.549302 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d3408d-d912-473f-a2b0-98ea776c48af None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:56.570847 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d3408d-d912-473f-a2b0-98ea776c48af None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:56.572989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d3408d-d912-473f-a2b0-98ea776c48af None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:56.592716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-d2d3408d-d912-473f-a2b0-98ea776c48af None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:46:56.592716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:46:56.592716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:46:56.592716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:46:56.592716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:46:56.592716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:46:56.592716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:46:56.592716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:46:56.592716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:46:56.592716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:46:56.592716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:46:56.592716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:46:56.592716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:46:56.592716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:46:56.592716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:46:56.592716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:46:56.594246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:46:56.594246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:46:56.594246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:46:56.594246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:46:56.594246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:46:56.594246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:46:56.594246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:46:56.594246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:46:56.594246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:46:56.594246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:46:56.594246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:46:56.594246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:46:56.594246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:46:56.594246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:46:56.594246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:46:56.594246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:46:56.594246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:46:56.595921 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:46:56.595921 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:46:56.595921 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:46:56.595921 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 624/1237] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:46:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:46:56.600774 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e986b207-e6e8-4ff9-846e-1159ab2361e7 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:56.724583 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e986b207-e6e8-4ff9-846e-1159ab2361e7 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:56.735265 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:56.763754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9dc3ba5-bebb-455c-a876-75f44f0ccc9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:56.764289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9dc3ba5-bebb-455c-a876-75f44f0ccc9c None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:56.764569 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9dc3ba5-bebb-455c-a876-75f44f0ccc9c None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:56.764845 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9dc3ba5-bebb-455c-a876-75f44f0ccc9c None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:56.765501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9dc3ba5-bebb-455c-a876-75f44f0ccc9c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:46:56.765814 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9dc3ba5-bebb-455c-a876-75f44f0ccc9c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:56.767223 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9dc3ba5-bebb-455c-a876-75f44f0ccc9c None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:56.780754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 614/1238] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:46:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:46:56.788845 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d3408d-d912-473f-a2b0-98ea776c48af None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:56.912034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d3408d-d912-473f-a2b0-98ea776c48af None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:56.955629 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c73e03-b775-41ef-b539-efbfd56e794b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:56.956317 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c73e03-b775-41ef-b539-efbfd56e794b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:56.956596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c73e03-b775-41ef-b539-efbfd56e794b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:56.957042 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c73e03-b775-41ef-b539-efbfd56e794b None admin] PATH_INFO: `/v3/users/placement` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:56.961200 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:56.968200 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c73e03-b775-41ef-b539-efbfd56e794b None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:56.970541 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c73e03-b775-41ef-b539-efbfd56e794b None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:56.990085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-87c73e03-b775-41ef-b539-efbfd56e794b None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 18 00:46:56.990085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:46:56.990085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:46:56.990085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:46:56.990085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:46:56.990085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:46:56.990085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:46:56.990085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:46:56.990085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:46:56.990085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:46:56.990085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:46:56.990085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:46:56.990085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:46:56.990085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:46:56.990085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:46:56.990085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:46:56.992037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:46:56.992037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:46:56.992037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:46:56.992037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:46:56.992037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:46:56.992037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:46:56.992037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:46:56.992037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:46:56.992037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:46:56.992037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:46:56.992037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:46:56.992037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:46:56.992037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:46:56.992037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:46:56.992037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:46:56.992037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:46:56.992037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:46:56.992037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:46:56.995056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:46:56.995056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:46:56.995056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:46:56.995056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:46:56.995056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:46:56.995056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:46:56.995056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:46:56.995056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:46:56.995056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 18 00:46:56.995056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:46:56.995056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 625/1239] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:46:56 2022] GET /identity/v3/users/placement => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:46:57.001295 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9dc3ba5-bebb-455c-a876-75f44f0ccc9c None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:57.143717 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9dc3ba5-bebb-455c-a876-75f44f0ccc9c None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:57.188864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14bfeb00-d6f2-41da-baa1-1833c5d23f81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:57.189451 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bfeb00-d6f2-41da-baa1-1833c5d23f81 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:57.189790 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bfeb00-d6f2-41da-baa1-1833c5d23f81 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:57.190058 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bfeb00-d6f2-41da-baa1-1833c5d23f81 None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:57.190829 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14bfeb00-d6f2-41da-baa1-1833c5d23f81 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:46:57.191132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14bfeb00-d6f2-41da-baa1-1833c5d23f81 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:57.192826 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14bfeb00-d6f2-41da-baa1-1833c5d23f81 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:57.226316 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 615/1240] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Sep 18 00:46:56 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:46:57.227781 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c73e03-b775-41ef-b539-efbfd56e794b None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:57.374681 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c73e03-b775-41ef-b539-efbfd56e794b None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:57.418017 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfd490b-7469-48ef-8530-02059602df4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:57.418645 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfd490b-7469-48ef-8530-02059602df4c None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:57.418930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfd490b-7469-48ef-8530-02059602df4c None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:57.419201 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfd490b-7469-48ef-8530-02059602df4c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:57.499492 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:57.512890 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfd490b-7469-48ef-8530-02059602df4c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:57.514396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfd490b-7469-48ef-8530-02059602df4c None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:57.658955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 626/1241] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:46:57 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 18 00:46:57.665331 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14bfeb00-d6f2-41da-baa1-1833c5d23f81 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:57.699160 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:57.803954 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14bfeb00-d6f2-41da-baa1-1833c5d23f81 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:57.860852 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758d04c2-5dc7-4be5-9de4-8f882f6742d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:57.862510 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758d04c2-5dc7-4be5-9de4-8f882f6742d9 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:57.862956 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758d04c2-5dc7-4be5-9de4-8f882f6742d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:57.863425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758d04c2-5dc7-4be5-9de4-8f882f6742d9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:57.881527 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-758d04c2-5dc7-4be5-9de4-8f882f6742d9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:57.883753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-758d04c2-5dc7-4be5-9de4-8f882f6742d9 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:57.903060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-758d04c2-5dc7-4be5-9de4-8f882f6742d9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:46:57.903060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:46:57.903060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:46:57.903060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:46:57.903060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:46:57.903060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:46:57.903060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:46:57.903060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:46:57.903060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:46:57.903060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:46:57.903060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:46:57.903060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:46:57.903060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:46:57.903060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:46:57.903060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:46:57.903060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:46:57.904586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:46:57.904586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:46:57.904586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:46:57.904586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:46:57.904586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:46:57.904586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:46:57.904586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:46:57.904586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:46:57.904586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:46:57.904586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:46:57.904586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:46:57.904586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:46:57.904586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:46:57.904586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:46:57.904586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:46:57.904586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:46:57.904586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:46:57.906189 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:46:57.906189 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:46:57.906189 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:46:57.906189 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 616/1242] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:46:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:46:57.911973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfd490b-7469-48ef-8530-02059602df4c None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:58.059111 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:58.073490 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfd490b-7469-48ef-8530-02059602df4c None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:58.116394 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35d69e0-af5f-428f-93df-df3ee98bc8bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:58.116989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35d69e0-af5f-428f-93df-df3ee98bc8bb None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:58.117266 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35d69e0-af5f-428f-93df-df3ee98bc8bb None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:58.117552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35d69e0-af5f-428f-93df-df3ee98bc8bb None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:58.118180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35d69e0-af5f-428f-93df-df3ee98bc8bb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:46:58.118511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35d69e0-af5f-428f-93df-df3ee98bc8bb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:58.119973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35d69e0-af5f-428f-93df-df3ee98bc8bb None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:58.132538 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 627/1243] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:46:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:46:58.139758 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758d04c2-5dc7-4be5-9de4-8f882f6742d9 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:58.253401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:46:58.278704 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758d04c2-5dc7-4be5-9de4-8f882f6742d9 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:58.322194 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6a5ed9-198c-4ade-a57a-16694fe16f6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:58.322835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6a5ed9-198c-4ade-a57a-16694fe16f6f None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:58.323104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6a5ed9-198c-4ade-a57a-16694fe16f6f None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:58.323372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6a5ed9-198c-4ade-a57a-16694fe16f6f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:58.331577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6a5ed9-198c-4ade-a57a-16694fe16f6f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:58.333229 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6a5ed9-198c-4ade-a57a-16694fe16f6f None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:58.346143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-df6a5ed9-198c-4ade-a57a-16694fe16f6f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:46:58.346143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:46:58.346143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:46:58.346143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:46:58.346143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:46:58.346143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:46:58.346143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:46:58.346143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:46:58.346143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:46:58.346143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:46:58.346143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:46:58.346143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:46:58.346143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:46:58.346143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:46:58.346143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:46:58.348216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:46:58.348216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:46:58.348216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:46:58.348216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:46:58.348216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:46:58.348216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:46:58.348216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:46:58.348216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:46:58.348216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:46:58.348216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:46:58.348216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:46:58.348216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:46:58.348216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:46:58.348216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:46:58.348216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:46:58.348216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:46:58.348216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:46:58.350196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:46:58.350196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:46:58.350196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:46:58.350196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:46:58.350196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:46:58.350196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:46:58.350196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:46:58.350196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:46:58.350196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 617/1244] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:46:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:46:58.355587 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35d69e0-af5f-428f-93df-df3ee98bc8bb None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:58.505703 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35d69e0-af5f-428f-93df-df3ee98bc8bb None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:58.567525 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d061e3-0f87-4d35-92d9-68b9086f02e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:58.568525 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d061e3-0f87-4d35-92d9-68b9086f02e1 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:58.568816 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d061e3-0f87-4d35-92d9-68b9086f02e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:58.569129 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d061e3-0f87-4d35-92d9-68b9086f02e1 None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:58.569857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d061e3-0f87-4d35-92d9-68b9086f02e1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:46:58.570181 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d061e3-0f87-4d35-92d9-68b9086f02e1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:58.572395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d061e3-0f87-4d35-92d9-68b9086f02e1 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:58.594540 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 628/1245] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:46:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:46:58.598539 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6a5ed9-198c-4ade-a57a-16694fe16f6f None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:46:58.742974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6a5ed9-198c-4ade-a57a-16694fe16f6f None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:46:58.793758 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a9b1f2-4240-489e-804a-0b43b59251bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:46:58.794470 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a9b1f2-4240-489e-804a-0b43b59251bd None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:46:58.794791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a9b1f2-4240-489e-804a-0b43b59251bd None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:46:58.795089 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a9b1f2-4240-489e-804a-0b43b59251bd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:46:58.796108 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a9b1f2-4240-489e-804a-0b43b59251bd None admin] RBAC: Adding query filter params (role.id=81523e544c7e4993af4d1e06ce3754bc, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=8f30116fb5ce43dab69efa8bafd9d049) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:46:58.796426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a9b1f2-4240-489e-804a-0b43b59251bd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:46:58.797917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a9b1f2-4240-489e-804a-0b43b59251bd None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:46:58.805261 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 618/1246] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:46:58 2022] GET /identity/v3/role_assignments?user.id=8f30116fb5ce43dab69efa8bafd9d049&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=81523e544c7e4993af4d1e06ce3754bc => generated 608 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:47:01.459408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adca627b-e300-4bc7-ae39-4d98098b8be9 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:01.459941 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adca627b-e300-4bc7-ae39-4d98098b8be9 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:01.459941 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adca627b-e300-4bc7-ae39-4d98098b8be9 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:01.461323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 629/1247] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:47:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:47:01.467366 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d27bf8c-9709-4a7e-8ee3-7dbce7529725 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:01.467892 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d27bf8c-9709-4a7e-8ee3-7dbce7529725 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:01.467892 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d27bf8c-9709-4a7e-8ee3-7dbce7529725 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:01.474549 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:01.494765 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-8d27bf8c-9709-4a7e-8ee3-7dbce7529725 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:47:01.496001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8d27bf8c-9709-4a7e-8ee3-7dbce7529725 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:47:01.574285 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 619/1248] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:47:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:47:01.887266 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adca627b-e300-4bc7-ae39-4d98098b8be9 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:01.907557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:02.021287 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adca627b-e300-4bc7-ae39-4d98098b8be9 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:02.072403 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea0e72f-ae9a-4fa9-842d-60da50685f38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:02.073409 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea0e72f-ae9a-4fa9-842d-60da50685f38 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:02.073882 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea0e72f-ae9a-4fa9-842d-60da50685f38 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:02.074337 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea0e72f-ae9a-4fa9-842d-60da50685f38 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:02.090715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea0e72f-ae9a-4fa9-842d-60da50685f38 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:47:02.099875 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-cea0e72f-ae9a-4fa9-842d-60da50685f38 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:47:02.099875 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:47:02.099875 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:47:02.099875 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:47:02.099875 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:47:02.099875 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:47:02.099875 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:47:02.099875 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:47:02.099875 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:47:02.099875 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:47:02.099875 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:47:02.099875 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:47:02.099875 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:47:02.099875 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:47:02.099875 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:47:02.099875 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:47:02.100853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:47:02.100853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:47:02.100853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:47:02.100853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:47:02.100853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:47:02.100853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:47:02.100853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:47:02.100853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:47:02.100853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:47:02.100853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:47:02.100853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:47:02.100853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:47:02.100853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:47:02.100853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:47:02.100853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:47:02.100853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:47:02.101688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:47:02.101688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:47:02.101688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:47:02.101688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:47:02.101688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:47:02.101688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:47:02.101688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:47:02.101688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:47:02.101688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:47:02.101688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:47:02.101688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:47:02.101688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:47:02.101688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 630/1249] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:47:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:47:02.108714 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d27bf8c-9709-4a7e-8ee3-7dbce7529725 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:02.216624 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d27bf8c-9709-4a7e-8ee3-7dbce7529725 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:02.247407 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:02.253116 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a866c0-48c0-45e4-9503-2f92c30adc47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:02.253730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a866c0-48c0-45e4-9503-2f92c30adc47 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:02.254024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a866c0-48c0-45e4-9503-2f92c30adc47 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:02.254278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a866c0-48c0-45e4-9503-2f92c30adc47 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:02.255028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a866c0-48c0-45e4-9503-2f92c30adc47 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:47:02.255369 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a866c0-48c0-45e4-9503-2f92c30adc47 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:02.256917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a866c0-48c0-45e4-9503-2f92c30adc47 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:02.266538 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 620/1250] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:47:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:47:02.273159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea0e72f-ae9a-4fa9-842d-60da50685f38 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:02.361962 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:02.397864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea0e72f-ae9a-4fa9-842d-60da50685f38 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:02.436934 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572e9976-41c3-48d2-a56f-2c488e8b6986 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:02.437577 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572e9976-41c3-48d2-a56f-2c488e8b6986 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:02.437851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572e9976-41c3-48d2-a56f-2c488e8b6986 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:02.438112 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572e9976-41c3-48d2-a56f-2c488e8b6986 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:02.448638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572e9976-41c3-48d2-a56f-2c488e8b6986 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:02.450194 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572e9976-41c3-48d2-a56f-2c488e8b6986 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:02.465832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-572e9976-41c3-48d2-a56f-2c488e8b6986 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:47:02.465832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:47:02.465832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:47:02.465832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:47:02.465832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:47:02.465832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:47:02.465832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:47:02.465832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:47:02.465832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:47:02.465832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:47:02.465832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:47:02.465832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:47:02.465832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:47:02.465832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:47:02.465832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:47:02.465832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:47:02.467573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:47:02.467573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:47:02.467573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:47:02.467573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:47:02.467573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:47:02.467573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:47:02.467573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:47:02.467573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:47:02.467573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:47:02.467573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:47:02.467573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:47:02.467573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:47:02.467573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:47:02.467573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:47:02.467573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:47:02.467573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:47:02.467573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:47:02.468483 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:47:02.468483 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:47:02.468483 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:47:02.468483 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 631/1251] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:47:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 18 00:47:02.473289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a866c0-48c0-45e4-9503-2f92c30adc47 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:02.586876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a866c0-48c0-45e4-9503-2f92c30adc47 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:02.625658 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfee831-7691-4d69-8e98-cd3621127ecf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:02.626658 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfee831-7691-4d69-8e98-cd3621127ecf None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:02.627112 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfee831-7691-4d69-8e98-cd3621127ecf None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:02.627484 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfee831-7691-4d69-8e98-cd3621127ecf None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:02.628453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfee831-7691-4d69-8e98-cd3621127ecf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:47:02.628831 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfee831-7691-4d69-8e98-cd3621127ecf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:02.630861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfee831-7691-4d69-8e98-cd3621127ecf None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:02.646713 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 621/1252] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:47:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:47:02.652886 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572e9976-41c3-48d2-a56f-2c488e8b6986 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:02.772694 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572e9976-41c3-48d2-a56f-2c488e8b6986 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:02.802903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:02.812475 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8d8b37-77e7-4fb9-9fe0-832f833c1c11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:02.813192 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8d8b37-77e7-4fb9-9fe0-832f833c1c11 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:02.813532 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8d8b37-77e7-4fb9-9fe0-832f833c1c11 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:02.813907 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8d8b37-77e7-4fb9-9fe0-832f833c1c11 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:02.823464 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8d8b37-77e7-4fb9-9fe0-832f833c1c11 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:02.824933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8d8b37-77e7-4fb9-9fe0-832f833c1c11 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:02.838306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-0d8d8b37-77e7-4fb9-9fe0-832f833c1c11 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 18 00:47:02.838306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:47:02.838306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:47:02.838306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:47:02.838306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:47:02.838306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:47:02.838306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:47:02.838306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:47:02.838306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:47:02.838306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:47:02.838306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:47:02.838306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:47:02.838306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:47:02.838306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:47:02.838306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:47:02.838306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:47:02.839396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:47:02.839396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:47:02.839396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:47:02.839396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:47:02.839396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:47:02.839396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:47:02.839396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:47:02.839396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:47:02.839396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:47:02.839396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:47:02.839396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:47:02.839396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:47:02.839396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:47:02.839396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:47:02.839396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:47:02.839396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:47:02.839396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:47:02.839396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:47:02.840669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:47:02.840669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:47:02.840669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:47:02.840669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:47:02.840669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:47:02.840669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:47:02.840669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:47:02.840669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:47:02.840669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 18 00:47:02.840669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:47:02.840669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 632/1253] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:47:02 2022] GET /identity/v3/users/placement => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:47:02.845956 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfee831-7691-4d69-8e98-cd3621127ecf None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:02.915702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:02.974948 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfee831-7691-4d69-8e98-cd3621127ecf None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:03.026906 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ced691-f6a9-414c-871d-9ffe013d106b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:03.027451 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ced691-f6a9-414c-871d-9ffe013d106b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:03.027727 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ced691-f6a9-414c-871d-9ffe013d106b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:03.028003 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ced691-f6a9-414c-871d-9ffe013d106b None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:03.028769 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ced691-f6a9-414c-871d-9ffe013d106b None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:47:03.029081 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ced691-f6a9-414c-871d-9ffe013d106b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:03.030616 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ced691-f6a9-414c-871d-9ffe013d106b None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:03.056934 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 622/1254] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Sep 18 00:47:02 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:47:03.058159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8d8b37-77e7-4fb9-9fe0-832f833c1c11 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:03.171229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8d8b37-77e7-4fb9-9fe0-832f833c1c11 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:03.207621 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d103605-8009-40d2-9872-b67f95b7294b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:03.208245 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d103605-8009-40d2-9872-b67f95b7294b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:03.208521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d103605-8009-40d2-9872-b67f95b7294b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:03.208791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d103605-8009-40d2-9872-b67f95b7294b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:03.270526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:03.284214 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d103605-8009-40d2-9872-b67f95b7294b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:03.285595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d103605-8009-40d2-9872-b67f95b7294b None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:03.401552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 633/1255] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:47:03 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 18 00:47:03.407534 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ced691-f6a9-414c-871d-9ffe013d106b None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:03.528394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ced691-f6a9-414c-871d-9ffe013d106b None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:03.544517 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:03.568113 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe805ab7-985d-4cdd-8fa1-ac2ef96bf3c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:03.568841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe805ab7-985d-4cdd-8fa1-ac2ef96bf3c6 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:03.569045 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe805ab7-985d-4cdd-8fa1-ac2ef96bf3c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:03.569317 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe805ab7-985d-4cdd-8fa1-ac2ef96bf3c6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:03.579877 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe805ab7-985d-4cdd-8fa1-ac2ef96bf3c6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:03.581396 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe805ab7-985d-4cdd-8fa1-ac2ef96bf3c6 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:03.601151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-fe805ab7-985d-4cdd-8fa1-ac2ef96bf3c6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:47:03.601151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:47:03.601151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:47:03.601151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:47:03.601151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:47:03.601151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:47:03.601151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:47:03.601151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:47:03.601151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:47:03.601151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:47:03.601151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:47:03.601151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:47:03.601151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:47:03.601151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:47:03.601151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:47:03.601151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:47:03.602154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:47:03.602154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:47:03.602154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:47:03.602154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:47:03.602154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:47:03.602154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:47:03.602154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:47:03.602154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:47:03.602154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:47:03.602154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:47:03.602154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:47:03.602154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:47:03.602154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:47:03.602154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:47:03.602154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:47:03.602154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:47:03.602154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:47:03.603675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:47:03.603675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:47:03.603675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:47:03.603675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 623/1256] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:47:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:47:03.610009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d103605-8009-40d2-9872-b67f95b7294b None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:03.697512 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:03.709871 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d103605-8009-40d2-9872-b67f95b7294b None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:03.744340 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9952c8f9-6400-4601-a93e-197a8e463ce8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:03.744940 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9952c8f9-6400-4601-a93e-197a8e463ce8 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:03.745219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9952c8f9-6400-4601-a93e-197a8e463ce8 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:03.745492 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9952c8f9-6400-4601-a93e-197a8e463ce8 None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:03.746127 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9952c8f9-6400-4601-a93e-197a8e463ce8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:47:03.746428 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9952c8f9-6400-4601-a93e-197a8e463ce8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:03.747838 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9952c8f9-6400-4601-a93e-197a8e463ce8 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:03.759599 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 634/1257] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:47:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:47:03.765703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe805ab7-985d-4cdd-8fa1-ac2ef96bf3c6 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:03.869137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe805ab7-985d-4cdd-8fa1-ac2ef96bf3c6 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:03.902194 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6c7ff6-cbbf-4476-8974-9d1410f25526 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:03.902841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6c7ff6-cbbf-4476-8974-9d1410f25526 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:03.903111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6c7ff6-cbbf-4476-8974-9d1410f25526 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:03.903377 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6c7ff6-cbbf-4476-8974-9d1410f25526 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:03.909870 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6c7ff6-cbbf-4476-8974-9d1410f25526 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:03.911321 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6c7ff6-cbbf-4476-8974-9d1410f25526 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:03.922224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-6d6c7ff6-cbbf-4476-8974-9d1410f25526 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:47:03.922224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:47:03.922224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:47:03.922224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:47:03.922224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:47:03.922224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:47:03.922224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:47:03.922224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:47:03.922224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:47:03.922224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:47:03.922224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:47:03.922224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:47:03.922224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:47:03.922224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:47:03.922224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:47:03.923406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:47:03.923406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:47:03.923406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:47:03.923406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:47:03.923406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:47:03.923406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:47:03.923406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:47:03.923406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:47:03.923406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:47:03.923406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:47:03.923406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:47:03.923406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:47:03.923406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:47:03.923406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:47:03.923406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:47:03.923406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:47:03.923406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:47:03.924323 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:47:03.924323 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:47:03.924323 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:47:03.924323 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:47:03.924323 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:47:03.924323 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:47:03.924323 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:47:03.924323 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:47:03.924323 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 624/1258] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:47:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:47:03.929299 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9952c8f9-6400-4601-a93e-197a8e463ce8 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:04.051542 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9952c8f9-6400-4601-a93e-197a8e463ce8 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:04.085577 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7d5e64-802d-416d-aa97-b5392a8e5051 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:04.086178 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7d5e64-802d-416d-aa97-b5392a8e5051 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:04.086479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7d5e64-802d-416d-aa97-b5392a8e5051 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:04.086749 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7d5e64-802d-416d-aa97-b5392a8e5051 None admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:04.087550 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7d5e64-802d-416d-aa97-b5392a8e5051 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:47:04.087854 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7d5e64-802d-416d-aa97-b5392a8e5051 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:04.089274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7d5e64-802d-416d-aa97-b5392a8e5051 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:04.100950 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 635/1259] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:47:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:47:04.107662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6c7ff6-cbbf-4476-8974-9d1410f25526 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:04.132693 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:04.208876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6c7ff6-cbbf-4476-8974-9d1410f25526 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:04.240757 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649d5cbe-aad3-4935-a647-44eade940e97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:04.241349 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649d5cbe-aad3-4935-a647-44eade940e97 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:04.241627 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649d5cbe-aad3-4935-a647-44eade940e97 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:04.241899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649d5cbe-aad3-4935-a647-44eade940e97 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:04.242651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649d5cbe-aad3-4935-a647-44eade940e97 None admin] RBAC: Adding query filter params (role.id=1e04c9fea65244d28e1e59fc4c64f32f, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=8f30116fb5ce43dab69efa8bafd9d049) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:47:04.243021 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649d5cbe-aad3-4935-a647-44eade940e97 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:04.244911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649d5cbe-aad3-4935-a647-44eade940e97 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:04.250138 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 625/1260] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:47:04 2022] GET /identity/v3/role_assignments?user.id=8f30116fb5ce43dab69efa8bafd9d049&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=1e04c9fea65244d28e1e59fc4c64f32f => generated 259 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:47:06.422652 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f02e3a-e39f-43e6-8056-2acdd7ab6848 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:06.423186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f02e3a-e39f-43e6-8056-2acdd7ab6848 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:06.423186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f02e3a-e39f-43e6-8056-2acdd7ab6848 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:06.424581 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 636/1261] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:47:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:47:06.429959 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b000d47-59cb-451e-9d5c-23ccd9823743 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:06.430386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b000d47-59cb-451e-9d5c-23ccd9823743 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:06.430529 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b000d47-59cb-451e-9d5c-23ccd9823743 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:06.450091 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-1b000d47-59cb-451e-9d5c-23ccd9823743 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:47:06.451351 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-1b000d47-59cb-451e-9d5c-23ccd9823743 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:47:06.522663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 626/1262] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:47:06 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:47:06.794162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f02e3a-e39f-43e6-8056-2acdd7ab6848 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:06.804936 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:06.898657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f02e3a-e39f-43e6-8056-2acdd7ab6848 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:06.932834 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae50bb2-d794-4966-b9d6-722a205b244e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:06.933459 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae50bb2-d794-4966-b9d6-722a205b244e None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:06.933768 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae50bb2-d794-4966-b9d6-722a205b244e None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:06.934043 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae50bb2-d794-4966-b9d6-722a205b244e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:06.947930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae50bb2-d794-4966-b9d6-722a205b244e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:47:06.954353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-0ae50bb2-d794-4966-b9d6-722a205b244e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:47:06.954353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:47:06.954353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:47:06.954353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:47:06.954353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:47:06.954353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:47:06.954353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:47:06.954353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:47:06.954353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:47:06.954353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:47:06.954353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:47:06.954353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:47:06.954353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:47:06.954353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:47:06.954353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:47:06.954353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:47:06.955509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:47:06.955509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:47:06.955509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:47:06.955509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:47:06.955509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:47:06.955509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:47:06.955509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:47:06.955509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:47:06.955509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:47:06.955509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:47:06.955509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:47:06.955509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:47:06.955509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:47:06.955509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:47:06.955509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:47:06.955509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:47:06.956408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:47:06.956408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:47:06.956408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:47:06.956408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:47:06.956408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:47:06.956408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:47:06.956408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:47:06.956408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:47:06.956408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:47:06.956408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:47:06.956408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:47:06.956408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:47:06.956408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 637/1263] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:47:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:47:06.963303 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b000d47-59cb-451e-9d5c-23ccd9823743 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:06.986038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:07.078137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b000d47-59cb-451e-9d5c-23ccd9823743 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:07.111193 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ad1f1d-2074-4026-bb4a-c196f1088c52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:07.111787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ad1f1d-2074-4026-bb4a-c196f1088c52 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:07.112061 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ad1f1d-2074-4026-bb4a-c196f1088c52 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:07.112329 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ad1f1d-2074-4026-bb4a-c196f1088c52 None admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:07.112990 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ad1f1d-2074-4026-bb4a-c196f1088c52 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:47:07.113289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ad1f1d-2074-4026-bb4a-c196f1088c52 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:07.114667 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ad1f1d-2074-4026-bb4a-c196f1088c52 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:07.122723 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 627/1264] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:47:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:47:07.128998 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae50bb2-d794-4966-b9d6-722a205b244e None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:07.196149 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:07.229165 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae50bb2-d794-4966-b9d6-722a205b244e None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:07.263470 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f289e89e-88f0-4e2a-98f6-f18fb607be1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:07.264026 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f289e89e-88f0-4e2a-98f6-f18fb607be1b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:07.264309 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f289e89e-88f0-4e2a-98f6-f18fb607be1b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:07.264585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f289e89e-88f0-4e2a-98f6-f18fb607be1b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:07.274682 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f289e89e-88f0-4e2a-98f6-f18fb607be1b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:07.276133 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f289e89e-88f0-4e2a-98f6-f18fb607be1b None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:07.290526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-f289e89e-88f0-4e2a-98f6-f18fb607be1b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:47:07.290526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:47:07.290526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:47:07.290526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:47:07.290526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:47:07.290526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:47:07.290526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:47:07.290526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:47:07.290526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:47:07.290526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:47:07.290526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:47:07.290526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:47:07.290526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:47:07.290526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:47:07.290526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:47:07.290526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:47:07.291690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:47:07.291690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:47:07.291690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:47:07.291690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:47:07.291690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:47:07.291690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:47:07.291690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:47:07.291690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:47:07.291690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:47:07.291690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:47:07.291690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:47:07.291690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:47:07.291690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:47:07.291690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:47:07.291690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:47:07.291690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:47:07.291690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:47:07.292714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:47:07.292714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:47:07.292714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:47:07.292714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 638/1265] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:47:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:47:07.297468 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ad1f1d-2074-4026-bb4a-c196f1088c52 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:07.378482 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:07.398156 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ad1f1d-2074-4026-bb4a-c196f1088c52 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:07.429962 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ef9484-bf78-4b51-b511-f9ebf2ec5915 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:07.431400 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ef9484-bf78-4b51-b511-f9ebf2ec5915 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:07.431639 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ef9484-bf78-4b51-b511-f9ebf2ec5915 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:07.431906 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ef9484-bf78-4b51-b511-f9ebf2ec5915 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:07.432543 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ef9484-bf78-4b51-b511-f9ebf2ec5915 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:47:07.432837 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ef9484-bf78-4b51-b511-f9ebf2ec5915 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:07.434177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ef9484-bf78-4b51-b511-f9ebf2ec5915 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:07.446049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 628/1266] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:47:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:47:07.455771 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f289e89e-88f0-4e2a-98f6-f18fb607be1b None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:07.580635 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f289e89e-88f0-4e2a-98f6-f18fb607be1b None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:07.601702 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:07.618893 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6a77b5-5846-4e4c-a9be-3368ca1256e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:07.619530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6a77b5-5846-4e4c-a9be-3368ca1256e5 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:07.619811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6a77b5-5846-4e4c-a9be-3368ca1256e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:07.620081 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6a77b5-5846-4e4c-a9be-3368ca1256e5 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:07.628867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6a77b5-5846-4e4c-a9be-3368ca1256e5 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:07.630251 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6a77b5-5846-4e4c-a9be-3368ca1256e5 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:07.642174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-1f6a77b5-5846-4e4c-a9be-3368ca1256e5 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 18 00:47:07.642174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:47:07.642174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:47:07.642174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:47:07.642174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:47:07.642174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:47:07.642174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:47:07.642174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:47:07.642174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:47:07.642174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:47:07.642174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:47:07.642174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:47:07.642174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:47:07.642174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:47:07.642174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:47:07.642174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:47:07.643350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:47:07.643350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:47:07.643350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:47:07.643350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:47:07.643350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:47:07.643350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:47:07.643350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:47:07.643350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:47:07.643350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:47:07.643350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:47:07.643350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:47:07.643350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:47:07.643350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:47:07.643350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:47:07.643350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:47:07.643350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:47:07.643350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:47:07.643350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:47:07.644503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:47:07.644503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:47:07.644503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:47:07.644503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:47:07.644503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:47:07.644503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:47:07.644503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:47:07.644503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:47:07.644503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 18 00:47:07.644503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:47:07.646608 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 639/1267] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:47:07 2022] GET /identity/v3/users/placement => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:47:07.649160 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ef9484-bf78-4b51-b511-f9ebf2ec5915 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:07.745903 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ef9484-bf78-4b51-b511-f9ebf2ec5915 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:07.777945 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e5384e-cf09-4fd1-8855-f7faace8f6c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:07.778567 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e5384e-cf09-4fd1-8855-f7faace8f6c2 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:07.778841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e5384e-cf09-4fd1-8855-f7faace8f6c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:07.779109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e5384e-cf09-4fd1-8855-f7faace8f6c2 None admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:07.779829 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e5384e-cf09-4fd1-8855-f7faace8f6c2 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:47:07.780128 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e5384e-cf09-4fd1-8855-f7faace8f6c2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:07.781530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e5384e-cf09-4fd1-8855-f7faace8f6c2 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:07.790571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:07.798928 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 629/1268] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Sep 18 00:47:07 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:47:07.804711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6a77b5-5846-4e4c-a9be-3368ca1256e5 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:07.907819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6a77b5-5846-4e4c-a9be-3368ca1256e5 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:07.941286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49cd7b4f-4eaa-4982-b842-4c3be6915c36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:07.941975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49cd7b4f-4eaa-4982-b842-4c3be6915c36 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:07.942308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49cd7b4f-4eaa-4982-b842-4c3be6915c36 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:07.942638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49cd7b4f-4eaa-4982-b842-4c3be6915c36 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:07.957266 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49cd7b4f-4eaa-4982-b842-4c3be6915c36 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:07.959174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49cd7b4f-4eaa-4982-b842-4c3be6915c36 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:07.978915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-49cd7b4f-4eaa-4982-b842-4c3be6915c36 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:47:07.978915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:47:07.978915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:47:07.978915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:47:07.978915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:47:07.978915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:47:07.978915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:47:07.978915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:47:07.978915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:47:07.978915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:47:07.978915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:47:07.978915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:47:07.978915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:47:07.978915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:47:07.978915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:47:07.978915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:47:07.980120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:47:07.980120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:47:07.980120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:47:07.980120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:47:07.980120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:47:07.980120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:47:07.980120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:47:07.980120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:47:07.980120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:47:07.980120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:47:07.980120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:47:07.980120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:47:07.980120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:47:07.980120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:47:07.980120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:47:07.980120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:47:07.980120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:47:07.981059 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:47:07.981059 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:47:07.981059 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:47:07.981059 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 640/1269] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:47:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:47:07.986735 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e5384e-cf09-4fd1-8855-f7faace8f6c2 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:08.086231 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e5384e-cf09-4fd1-8855-f7faace8f6c2 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:08.120308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e09a2a8-603e-44de-afab-1231faf4ff6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:08.120908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e09a2a8-603e-44de-afab-1231faf4ff6c None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:08.121187 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e09a2a8-603e-44de-afab-1231faf4ff6c None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:08.121461 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e09a2a8-603e-44de-afab-1231faf4ff6c None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:08.122105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e09a2a8-603e-44de-afab-1231faf4ff6c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:47:08.122406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e09a2a8-603e-44de-afab-1231faf4ff6c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:08.123793 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e09a2a8-603e-44de-afab-1231faf4ff6c None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:08.134988 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 630/1270] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:47:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:47:08.140663 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49cd7b4f-4eaa-4982-b842-4c3be6915c36 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:08.172463 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:08.241500 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49cd7b4f-4eaa-4982-b842-4c3be6915c36 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:08.273329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b430919-316d-4518-94f1-27d817345865 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:08.273949 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b430919-316d-4518-94f1-27d817345865 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:08.274222 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b430919-316d-4518-94f1-27d817345865 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:08.275049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b430919-316d-4518-94f1-27d817345865 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:08.281420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b430919-316d-4518-94f1-27d817345865 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:08.282892 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b430919-316d-4518-94f1-27d817345865 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:08.294440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-7b430919-316d-4518-94f1-27d817345865 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:47:08.294440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:47:08.294440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:47:08.294440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:47:08.294440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:47:08.294440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:47:08.294440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:47:08.294440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:47:08.294440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:47:08.294440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:47:08.294440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:47:08.294440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:47:08.294440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:47:08.294440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:47:08.294440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:47:08.295612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:47:08.295612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:47:08.295612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:47:08.295612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:47:08.295612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:47:08.295612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:47:08.295612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:47:08.295612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:47:08.295612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:47:08.295612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:47:08.295612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:47:08.295612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:47:08.295612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:47:08.295612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:47:08.295612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:47:08.295612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:47:08.295612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:47:08.296668 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:47:08.296668 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:47:08.296668 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:47:08.296668 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:47:08.296668 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:47:08.296668 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:47:08.296668 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:47:08.296668 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:47:08.296668 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 641/1271] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:47:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:47:08.301470 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e09a2a8-603e-44de-afab-1231faf4ff6c None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:08.359365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:08.401861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e09a2a8-603e-44de-afab-1231faf4ff6c None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:08.434742 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6977d5e-62fa-494a-9071-c368a1b9da3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:08.435377 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6977d5e-62fa-494a-9071-c368a1b9da3b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:08.435534 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6977d5e-62fa-494a-9071-c368a1b9da3b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:08.435804 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6977d5e-62fa-494a-9071-c368a1b9da3b None admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:08.436496 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6977d5e-62fa-494a-9071-c368a1b9da3b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:47:08.436802 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6977d5e-62fa-494a-9071-c368a1b9da3b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:08.438215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6977d5e-62fa-494a-9071-c368a1b9da3b None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:08.453681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 631/1272] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:47:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:47:08.459796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b430919-316d-4518-94f1-27d817345865 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:08.552491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:08.561841 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b430919-316d-4518-94f1-27d817345865 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:08.595231 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e6eefc-1fed-43ef-bfae-f12095a4586a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:08.595890 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e6eefc-1fed-43ef-bfae-f12095a4586a None admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:08.596168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e6eefc-1fed-43ef-bfae-f12095a4586a None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:08.596446 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e6eefc-1fed-43ef-bfae-f12095a4586a None admin] PATH_INFO: `/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/8f30116fb5ce43dab69efa8bafd9d049/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:08.598397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e6eefc-1fed-43ef-bfae-f12095a4586a None admin] RBAC: Authorizing `identity:create_grant(project_id=c520a8da3dc54adba4f7cc04ef42e367, user_id=8f30116fb5ce43dab69efa8bafd9d049, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:08.599852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e6eefc-1fed-43ef-bfae-f12095a4586a None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:08.612418 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 642/1273] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 18 00:47:08 2022] PUT /identity/v3/projects/c520a8da3dc54adba4f7cc04ef42e367/users/8f30116fb5ce43dab69efa8bafd9d049/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:47:10.776211 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4be8df-35c9-46fd-a598-f645ced56027 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:10.776706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4be8df-35c9-46fd-a598-f645ced56027 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:10.776706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4be8df-35c9-46fd-a598-f645ced56027 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:10.777995 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 632/1274] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:47:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:47:10.783340 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bad2d68-72ba-4529-a106-91cc965a8da0 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:10.783752 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bad2d68-72ba-4529-a106-91cc965a8da0 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:10.783898 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bad2d68-72ba-4529-a106-91cc965a8da0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:10.802264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-6bad2d68-72ba-4529-a106-91cc965a8da0 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:47:10.803507 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-6bad2d68-72ba-4529-a106-91cc965a8da0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:47:10.867632 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 643/1275] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:47:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:47:11.155058 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4be8df-35c9-46fd-a598-f645ced56027 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:11.255202 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4be8df-35c9-46fd-a598-f645ced56027 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:11.272998 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:11.289160 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758c4d5f-a0fd-4fb7-befd-8fe8fa4ad7ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:11.289784 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758c4d5f-a0fd-4fb7-befd-8fe8fa4ad7ac None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:11.290574 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758c4d5f-a0fd-4fb7-befd-8fe8fa4ad7ac None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:11.290844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758c4d5f-a0fd-4fb7-befd-8fe8fa4ad7ac None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:11.300656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-758c4d5f-a0fd-4fb7-befd-8fe8fa4ad7ac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 18 00:47:11.305401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-758c4d5f-a0fd-4fb7-befd-8fe8fa4ad7ac None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:47:11.305401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:47:11.305401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:47:11.305401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:47:11.305401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:47:11.305401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:47:11.305401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:47:11.305401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:47:11.305401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:47:11.305401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:47:11.305401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:47:11.305401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:47:11.305401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 18 00:47:11.305401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 18 00:47:11.305401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 18 00:47:11.305401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 18 00:47:11.307359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 18 00:47:11.307359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 18 00:47:11.307359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 18 00:47:11.307359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 18 00:47:11.307359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:47:11.307359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:47:11.307359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:47:11.307359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:47:11.307359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:47:11.307359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:47:11.307359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:47:11.307359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:47:11.307359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:47:11.307359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:47:11.307359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:47:11.307359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:47:11.308395 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:47:11.308395 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:47:11.308395 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:47:11.308395 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:47:11.308395 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 18 00:47:11.308395 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 18 00:47:11.308395 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 18 00:47:11.308395 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 18 00:47:11.308395 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 18 00:47:11.308395 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 18 00:47:11.308395 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 18 00:47:11.308395 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:47:11.308395 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 633/1276] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 18 00:47:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:47:11.313421 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bad2d68-72ba-4529-a106-91cc965a8da0 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:11.392174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:11.410034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bad2d68-72ba-4529-a106-91cc965a8da0 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:11.444379 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320347c2-54be-448e-b837-5f5f5375c372 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:11.444983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320347c2-54be-448e-b837-5f5f5375c372 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:11.445261 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320347c2-54be-448e-b837-5f5f5375c372 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:11.445532 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320347c2-54be-448e-b837-5f5f5375c372 None admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:11.446188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320347c2-54be-448e-b837-5f5f5375c372 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:47:11.447056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320347c2-54be-448e-b837-5f5f5375c372 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:11.448430 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320347c2-54be-448e-b837-5f5f5375c372 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:11.456138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 644/1277] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 18 00:47:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:47:11.462667 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758c4d5f-a0fd-4fb7-befd-8fe8fa4ad7ac None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:11.560634 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758c4d5f-a0fd-4fb7-befd-8fe8fa4ad7ac None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:11.594404 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7ec6bd-3a6b-4b70-a52c-1b2ce8284c0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:11.595060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7ec6bd-3a6b-4b70-a52c-1b2ce8284c0b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:11.595338 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7ec6bd-3a6b-4b70-a52c-1b2ce8284c0b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:11.595612 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7ec6bd-3a6b-4b70-a52c-1b2ce8284c0b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:11.606038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7ec6bd-3a6b-4b70-a52c-1b2ce8284c0b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:11.607507 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7ec6bd-3a6b-4b70-a52c-1b2ce8284c0b None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:11.623737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-5e7ec6bd-3a6b-4b70-a52c-1b2ce8284c0b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:47:11.623737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:47:11.623737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:47:11.623737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:47:11.623737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:47:11.623737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:47:11.623737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:47:11.623737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:47:11.623737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:47:11.623737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:47:11.623737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:47:11.623737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:47:11.623737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:47:11.623737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:47:11.623737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:47:11.623737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:47:11.624836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:47:11.624836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:47:11.624836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:47:11.624836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:47:11.624836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:47:11.624836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:47:11.624836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:47:11.624836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:47:11.624836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:47:11.624836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:47:11.624836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:47:11.624836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:47:11.624836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:47:11.624836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:47:11.624836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:47:11.624836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:47:11.624836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:47:11.626057 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:47:11.626057 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:47:11.626057 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:47:11.626057 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 634/1278] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:47:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:47:11.631163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320347c2-54be-448e-b837-5f5f5375c372 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:11.728644 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320347c2-54be-448e-b837-5f5f5375c372 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:11.760720 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7f302d-aa57-476a-8d6e-375d77e9dc2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:11.761315 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7f302d-aa57-476a-8d6e-375d77e9dc2b None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:11.761590 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7f302d-aa57-476a-8d6e-375d77e9dc2b None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:11.761857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7f302d-aa57-476a-8d6e-375d77e9dc2b None admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:11.762509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7f302d-aa57-476a-8d6e-375d77e9dc2b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:47:11.762807 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7f302d-aa57-476a-8d6e-375d77e9dc2b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:11.764297 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7f302d-aa57-476a-8d6e-375d77e9dc2b None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:11.776137 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 645/1279] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:47:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:47:11.781855 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7ec6bd-3a6b-4b70-a52c-1b2ce8284c0b None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:11.812317 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:11.886667 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7ec6bd-3a6b-4b70-a52c-1b2ce8284c0b None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:11.923334 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7745f0b1-2d20-4ddd-9a0f-b43a38e22413 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:11.923959 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7745f0b1-2d20-4ddd-9a0f-b43a38e22413 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:11.924236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7745f0b1-2d20-4ddd-9a0f-b43a38e22413 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:11.924506 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7745f0b1-2d20-4ddd-9a0f-b43a38e22413 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:11.932537 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7745f0b1-2d20-4ddd-9a0f-b43a38e22413 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:11.934001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7745f0b1-2d20-4ddd-9a0f-b43a38e22413 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:11.946250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-7745f0b1-2d20-4ddd-9a0f-b43a38e22413 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 18 00:47:11.946250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:47:11.946250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:47:11.946250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:47:11.946250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:47:11.946250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:47:11.946250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:47:11.946250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:47:11.946250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:47:11.946250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:47:11.946250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:47:11.946250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:47:11.946250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 18 00:47:11.946250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 18 00:47:11.946250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 18 00:47:11.946250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 18 00:47:11.948289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:47:11.948289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:47:11.948289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 18 00:47:11.948289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:47:11.948289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 18 00:47:11.948289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 18 00:47:11.948289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:47:11.948289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:47:11.948289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:47:11.948289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:47:11.948289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:47:11.948289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:47:11.948289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:47:11.948289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:47:11.948289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:47:11.948289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:47:11.948289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:47:11.948289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:47:11.949472 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:47:11.949472 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:47:11.949472 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 18 00:47:11.949472 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 18 00:47:11.949472 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 18 00:47:11.949472 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 18 00:47:11.949472 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 18 00:47:11.949472 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 18 00:47:11.949472 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 18 00:47:11.949472 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:47:11.949472 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 635/1280] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:47:11 2022] GET /identity/v3/users/placement => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:47:11.954143 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7f302d-aa57-476a-8d6e-375d77e9dc2b None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:11.958002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:12.069178 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7f302d-aa57-476a-8d6e-375d77e9dc2b None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:12.106680 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2638d664-f012-4218-a5c9-703c943ef036 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:12.107293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2638d664-f012-4218-a5c9-703c943ef036 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:12.107572 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2638d664-f012-4218-a5c9-703c943ef036 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:12.107844 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2638d664-f012-4218-a5c9-703c943ef036 None admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:12.108583 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2638d664-f012-4218-a5c9-703c943ef036 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:47:12.108883 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2638d664-f012-4218-a5c9-703c943ef036 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:12.110322 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2638d664-f012-4218-a5c9-703c943ef036 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:12.129533 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 646/1281] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Sep 18 00:47:11 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:47:12.135579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7745f0b1-2d20-4ddd-9a0f-b43a38e22413 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:12.229019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:12.239059 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7745f0b1-2d20-4ddd-9a0f-b43a38e22413 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:12.271589 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c47ed48-d987-44dc-a04f-e1217e582071 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:12.272197 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c47ed48-d987-44dc-a04f-e1217e582071 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:12.272477 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c47ed48-d987-44dc-a04f-e1217e582071 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:12.272749 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c47ed48-d987-44dc-a04f-e1217e582071 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:12.337231 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c47ed48-d987-44dc-a04f-e1217e582071 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:12.338624 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c47ed48-d987-44dc-a04f-e1217e582071 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:12.427446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:12.438223 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 636/1282] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 18 00:47:12 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 18 00:47:12.444067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2638d664-f012-4218-a5c9-703c943ef036 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:12.506128 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:12.545426 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2638d664-f012-4218-a5c9-703c943ef036 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:12.578562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93bb6f24-415b-49ce-b267-28876c751fc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:12.579185 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bb6f24-415b-49ce-b267-28876c751fc3 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:12.579467 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bb6f24-415b-49ce-b267-28876c751fc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:12.579743 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bb6f24-415b-49ce-b267-28876c751fc3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:12.590009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93bb6f24-415b-49ce-b267-28876c751fc3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:12.591425 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93bb6f24-415b-49ce-b267-28876c751fc3 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:12.606225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-93bb6f24-415b-49ce-b267-28876c751fc3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:47:12.606225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:47:12.606225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:47:12.606225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:47:12.606225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:47:12.606225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:47:12.606225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:47:12.606225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:47:12.606225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:47:12.606225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:47:12.606225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:47:12.606225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:47:12.606225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 18 00:47:12.606225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 18 00:47:12.606225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 18 00:47:12.606225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 18 00:47:12.607427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:47:12.607427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:47:12.607427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:47:12.607427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:47:12.607427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:47:12.607427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:47:12.607427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:47:12.607427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:47:12.607427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:47:12.607427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:47:12.607427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:47:12.607427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:47:12.607427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:47:12.607427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:47:12.607427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:47:12.607427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:47:12.607427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 18 00:47:12.608576 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 18 00:47:12.608576 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 18 00:47:12.608576 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:47:12.608576 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 647/1283] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:47:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:47:12.613107 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c47ed48-d987-44dc-a04f-e1217e582071 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:12.707306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c47ed48-d987-44dc-a04f-e1217e582071 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:12.742033 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4205669c-04d8-4497-bd1f-f9753340eea2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:12.742659 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4205669c-04d8-4497-bd1f-f9753340eea2 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:12.742936 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4205669c-04d8-4497-bd1f-f9753340eea2 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:12.743207 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4205669c-04d8-4497-bd1f-f9753340eea2 None admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:12.743844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4205669c-04d8-4497-bd1f-f9753340eea2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:47:12.744146 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4205669c-04d8-4497-bd1f-f9753340eea2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:12.745508 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4205669c-04d8-4497-bd1f-f9753340eea2 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:12.756483 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 637/1284] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 18 00:47:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:47:12.762177 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93bb6f24-415b-49ce-b267-28876c751fc3 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:12.859933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93bb6f24-415b-49ce-b267-28876c751fc3 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:12.873315 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:12.893248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8771b2a-1f85-4db9-823a-9b4cbae4bfb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:12.893901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8771b2a-1f85-4db9-823a-9b4cbae4bfb5 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:12.894177 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8771b2a-1f85-4db9-823a-9b4cbae4bfb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:12.894448 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8771b2a-1f85-4db9-823a-9b4cbae4bfb5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:12.904228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8771b2a-1f85-4db9-823a-9b4cbae4bfb5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:12.905703 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8771b2a-1f85-4db9-823a-9b4cbae4bfb5 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:12.918575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application [None req-a8771b2a-1f85-4db9-823a-9b4cbae4bfb5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:47:12.918575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:47:12.918575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:47:12.918575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:47:12.918575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:47:12.918575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:47:12.918575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:47:12.918575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:47:12.918575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:47:12.918575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:47:12.918575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:47:12.918575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:47:12.918575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 18 00:47:12.918575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 18 00:47:12.918575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 18 00:47:12.919604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 18 00:47:12.919604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:47:12.919604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:47:12.919604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:47:12.919604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:47:12.919604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:47:12.919604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:47:12.919604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:47:12.919604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application with Lock( Sep 18 00:47:12.919604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:47:12.919604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:47:12.919604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:47:12.919604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:47:12.919604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:47:12.919604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:47:12.919604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:47:12.919604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:47:12.920708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 18 00:47:12.920708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 18 00:47:12.920708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 18 00:47:12.920708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 18 00:47:12.920708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 18 00:47:12.920708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 18 00:47:12.920708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 18 00:47:12.920708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: ERROR keystone.server.flask.application Sep 18 00:47:12.920708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 648/1285] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 18 00:47:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:47:12.925565 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4205669c-04d8-4497-bd1f-f9753340eea2 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:12.976841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:13.026000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4205669c-04d8-4497-bd1f-f9753340eea2 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:13.058804 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0b3176-d860-4a96-b6b5-086da4d7730c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:13.059394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0b3176-d860-4a96-b6b5-086da4d7730c None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:13.059670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0b3176-d860-4a96-b6b5-086da4d7730c None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:13.059941 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0b3176-d860-4a96-b6b5-086da4d7730c None admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:13.060627 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0b3176-d860-4a96-b6b5-086da4d7730c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:47:13.060928 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0b3176-d860-4a96-b6b5-086da4d7730c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:13.062330 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0b3176-d860-4a96-b6b5-086da4d7730c None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:13.073693 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 638/1286] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 18 00:47:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:47:13.079799 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8771b2a-1f85-4db9-823a-9b4cbae4bfb5 None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:13.174917 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8771b2a-1f85-4db9-823a-9b4cbae4bfb5 None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:13.206394 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edecbf9c-66e7-4c5a-af0c-c14f75169274 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:13.207010 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edecbf9c-66e7-4c5a-af0c-c14f75169274 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:13.207283 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edecbf9c-66e7-4c5a-af0c-c14f75169274 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:13.207551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edecbf9c-66e7-4c5a-af0c-c14f75169274 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:13.208262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edecbf9c-66e7-4c5a-af0c-c14f75169274 None admin] RBAC: Adding query filter params (role.id=1e04c9fea65244d28e1e59fc4c64f32f, scope.project.id=c520a8da3dc54adba4f7cc04ef42e367, user.id=8f30116fb5ce43dab69efa8bafd9d049) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:47:13.208565 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edecbf9c-66e7-4c5a-af0c-c14f75169274 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:13.209979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edecbf9c-66e7-4c5a-af0c-c14f75169274 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:13.216126 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 649/1287] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 18 00:47:13 2022] GET /identity/v3/role_assignments?user.id=8f30116fb5ce43dab69efa8bafd9d049&scope.project.id=c520a8da3dc54adba4f7cc04ef42e367&role.id=1e04c9fea65244d28e1e59fc4c64f32f => generated 608 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:47:15.413307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14a39b4-97f8-4ebe-ae41-3bd311039ba8 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:15.413867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14a39b4-97f8-4ebe-ae41-3bd311039ba8 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:15.413867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14a39b4-97f8-4ebe-ae41-3bd311039ba8 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:15.415733 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 639/1288] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:47:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:47:15.421538 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcc4c38-e6c4-44ad-b89f-1eef7c5aec8c None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:15.422005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcc4c38-e6c4-44ad-b89f-1eef7c5aec8c None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:15.422104 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcc4c38-e6c4-44ad-b89f-1eef7c5aec8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:15.441945 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-5dcc4c38-e6c4-44ad-b89f-1eef7c5aec8c None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:47:15.443200 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-5dcc4c38-e6c4-44ad-b89f-1eef7c5aec8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:47:15.472575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:15.510741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 650/1289] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:47:15 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 18 00:47:15.789742 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e14a39b4-97f8-4ebe-ae41-3bd311039ba8 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:15.886688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e14a39b4-97f8-4ebe-ae41-3bd311039ba8 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:15.897023 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:15.920513 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6a5c9c-2c95-47ec-877c-4c95b9ce1672 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:15.921134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6a5c9c-2c95-47ec-877c-4c95b9ce1672 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:15.921408 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6a5c9c-2c95-47ec-877c-4c95b9ce1672 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:15.921675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6a5c9c-2c95-47ec-877c-4c95b9ce1672 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:15.922249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa6a5c9c-2c95-47ec-877c-4c95b9ce1672 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:15.923634 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa6a5c9c-2c95-47ec-877c-4c95b9ce1672 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:15.932635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-fa6a5c9c-2c95-47ec-877c-4c95b9ce1672 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 18 00:47:15.932635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:47:15.932635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:47:15.932635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:47:15.932635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:47:15.932635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:47:15.932635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:47:15.932635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:47:15.932635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 18 00:47:15.932635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:47:15.932635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:47:15.932635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:47:15.932635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:47:15.932635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:47:15.932635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:47:15.932635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:47:15.932635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:47:15.934307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:47:15.934307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:47:15.934307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:47:15.934307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:47:15.934307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:47:15.934307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:47:15.934307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:47:15.934307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:47:15.934307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:47:15.934307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:47:15.934307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:47:15.934307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:47:15.934307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:47:15.934307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:47:15.934307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:47:15.934307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:47:15.936460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:47:15.936460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:47:15.936460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:47:15.936460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:47:15.936460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:47:15.936460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:47:15.936460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:47:15.936460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:47:15.936460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:47:15.936460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:47:15.936460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:47:15.936460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:47:15.936460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 18 00:47:15.936460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:47:15.936460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 640/1290] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Sep 18 00:47:15 2022] GET /identity/v3/services/placement => generated 90 bytes in 145 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:47:15.940679 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dcc4c38-e6c4-44ad-b89f-1eef7c5aec8c None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:16.058109 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dcc4c38-e6c4-44ad-b89f-1eef7c5aec8c None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:16.093083 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751e5020-7500-4184-a3cd-4b943a5f3a34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:16.093799 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751e5020-7500-4184-a3cd-4b943a5f3a34 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:16.094076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751e5020-7500-4184-a3cd-4b943a5f3a34 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:16.094343 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751e5020-7500-4184-a3cd-4b943a5f3a34 None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:16.094995 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751e5020-7500-4184-a3cd-4b943a5f3a34 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:47:16.095382 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751e5020-7500-4184-a3cd-4b943a5f3a34 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:16.097053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751e5020-7500-4184-a3cd-4b943a5f3a34 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:16.101760 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 651/1291] 10.222.0.42 () {56 vars in 1116 bytes} [Sun Sep 18 00:47:15 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:47:16.108197 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6a5c9c-2c95-47ec-877c-4c95b9ce1672 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:16.203604 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6a5c9c-2c95-47ec-877c-4c95b9ce1672 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:16.235841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24bb04ce-ad2e-4dab-a2a2-729ac1e7ee38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:16.236431 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bb04ce-ad2e-4dab-a2a2-729ac1e7ee38 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:16.236703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bb04ce-ad2e-4dab-a2a2-729ac1e7ee38 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:16.236969 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bb04ce-ad2e-4dab-a2a2-729ac1e7ee38 None admin] PATH_INFO: `/v3/services` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:16.237591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24bb04ce-ad2e-4dab-a2a2-729ac1e7ee38 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:47:16.237889 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24bb04ce-ad2e-4dab-a2a2-729ac1e7ee38 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:16.239256 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24bb04ce-ad2e-4dab-a2a2-729ac1e7ee38 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:16.243535 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 641/1292] 10.222.0.42 () {56 vars in 1116 bytes} [Sun Sep 18 00:47:16 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:47:18.410020 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52a6df4-9f5b-4720-8db0-aac77a083232 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:18.410614 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52a6df4-9f5b-4720-8db0-aac77a083232 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:18.410614 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52a6df4-9f5b-4720-8db0-aac77a083232 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:18.411908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 652/1293] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:47:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:47:18.417752 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298f00c6-0d50-42ac-9d37-294fdaa74498 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:18.418186 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298f00c6-0d50-42ac-9d37-294fdaa74498 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:18.418289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298f00c6-0d50-42ac-9d37-294fdaa74498 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:18.437042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-298f00c6-0d50-42ac-9d37-294fdaa74498 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:47:18.438242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-298f00c6-0d50-42ac-9d37-294fdaa74498 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:47:18.467077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:18.504162 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 642/1294] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:47:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:47:18.776445 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52a6df4-9f5b-4720-8db0-aac77a083232 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:18.806301 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:18.874650 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52a6df4-9f5b-4720-8db0-aac77a083232 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:18.908186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a3f123-91c4-400f-a20e-7d1dc03566e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:18.908785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a3f123-91c4-400f-a20e-7d1dc03566e0 None admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:18.909065 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a3f123-91c4-400f-a20e-7d1dc03566e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:18.909365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a3f123-91c4-400f-a20e-7d1dc03566e0 None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:18.910011 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a3f123-91c4-400f-a20e-7d1dc03566e0 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:18.911400 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a3f123-91c4-400f-a20e-7d1dc03566e0 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:18.925639 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 653/1295] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 18 00:47:18 2022] POST /identity/v3/services => generated 246 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:47:21.099903 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e93713a-bf64-4d3f-a9bc-51711e675ef2 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:21.100464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e93713a-bf64-4d3f-a9bc-51711e675ef2 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:21.100464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e93713a-bf64-4d3f-a9bc-51711e675ef2 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:21.101781 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 643/1296] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:47:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:47:21.107768 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e73a3c9-7bbe-46b3-bf0e-0573264c69d8 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:21.108191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e73a3c9-7bbe-46b3-bf0e-0573264c69d8 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:21.108312 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e73a3c9-7bbe-46b3-bf0e-0573264c69d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:21.126891 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-3e73a3c9-7bbe-46b3-bf0e-0573264c69d8 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:47:21.128079 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3e73a3c9-7bbe-46b3-bf0e-0573264c69d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:47:21.206094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 654/1297] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:47:21 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:47:21.477342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e93713a-bf64-4d3f-a9bc-51711e675ef2 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:21.584671 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e93713a-bf64-4d3f-a9bc-51711e675ef2 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:21.619154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c8d0d5-7e5d-45cd-b1bf-18487fd8c06b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:21.619779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c8d0d5-7e5d-45cd-b1bf-18487fd8c06b None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:21.620054 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c8d0d5-7e5d-45cd-b1bf-18487fd8c06b None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:21.620322 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c8d0d5-7e5d-45cd-b1bf-18487fd8c06b None admin] PATH_INFO: `/v3/services/placement` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:21.620904 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c8d0d5-7e5d-45cd-b1bf-18487fd8c06b None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:21.622261 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c8d0d5-7e5d-45cd-b1bf-18487fd8c06b None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:21.630966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-38c8d0d5-7e5d-45cd-b1bf-18487fd8c06b None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 18 00:47:21.630966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:47:21.630966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:47:21.630966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:47:21.630966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:47:21.630966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:47:21.630966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:47:21.630966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:47:21.630966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 18 00:47:21.630966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:47:21.630966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:47:21.630966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:47:21.630966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:47:21.630966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:47:21.630966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:47:21.630966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:47:21.630966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:47:21.632106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:47:21.632106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:47:21.632106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:47:21.632106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:47:21.632106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:47:21.632106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:47:21.632106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:47:21.632106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:47:21.632106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:47:21.632106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:47:21.632106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:47:21.632106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:47:21.632106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:47:21.632106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:47:21.632106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:47:21.632106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:47:21.633077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:47:21.633077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:47:21.633077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:47:21.633077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:47:21.633077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:47:21.633077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:47:21.633077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:47:21.633077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:47:21.633077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:47:21.633077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:47:21.633077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:47:21.633077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:47:21.633077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 18 00:47:21.633077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:47:21.633077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 644/1298] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Sep 18 00:47:21 2022] GET /identity/v3/services/placement => generated 90 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:47:21.638761 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e73a3c9-7bbe-46b3-bf0e-0573264c69d8 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:21.658098 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:21.749667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e73a3c9-7bbe-46b3-bf0e-0573264c69d8 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:21.790108 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d153af8-3ef2-4c8a-9752-b6a56eb4c0ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:21.791270 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d153af8-3ef2-4c8a-9752-b6a56eb4c0ad None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:21.791552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d153af8-3ef2-4c8a-9752-b6a56eb4c0ad None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:21.791823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d153af8-3ef2-4c8a-9752-b6a56eb4c0ad None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:21.792463 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d153af8-3ef2-4c8a-9752-b6a56eb4c0ad None admin] RBAC: Adding query filter params (name=placement) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:47:21.792770 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d153af8-3ef2-4c8a-9752-b6a56eb4c0ad None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:21.794164 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d153af8-3ef2-4c8a-9752-b6a56eb4c0ad None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:21.799641 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 655/1299] 10.222.0.42 () {56 vars in 1116 bytes} [Sun Sep 18 00:47:21 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:47:21.805777 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c8d0d5-7e5d-45cd-b1bf-18487fd8c06b None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:21.839448 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:21.917515 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c8d0d5-7e5d-45cd-b1bf-18487fd8c06b None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:21.952768 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3b3913-8e99-490d-b99a-96b4df387fb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:21.953408 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3b3913-8e99-490d-b99a-96b4df387fb9 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:21.953680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3b3913-8e99-490d-b99a-96b4df387fb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:21.953949 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3b3913-8e99-490d-b99a-96b4df387fb9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:21.954687 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3b3913-8e99-490d-b99a-96b4df387fb9 None admin] RBAC: Adding query filter params (interface=public, service_id=991a01d101cc4c8fa816448265ab8f91, region_id=RegionOne) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:47:21.954994 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3b3913-8e99-490d-b99a-96b4df387fb9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:21.956354 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3b3913-8e99-490d-b99a-96b4df387fb9 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:21.961761 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 645/1300] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Sep 18 00:47:21 2022] GET /identity/v3/endpoints?service_id=991a01d101cc4c8fa816448265ab8f91&interface=public®ion_id=RegionOne => generated 194 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:47:21.968419 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d153af8-3ef2-4c8a-9752-b6a56eb4c0ad None admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:22.054337 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:22.075587 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d153af8-3ef2-4c8a-9752-b6a56eb4c0ad None admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:22.109664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2e1f61-2db7-491c-96ef-7af2f856ac10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:22.110321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2e1f61-2db7-491c-96ef-7af2f856ac10 None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:22.110581 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2e1f61-2db7-491c-96ef-7af2f856ac10 None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:22.110853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2e1f61-2db7-491c-96ef-7af2f856ac10 None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:22.111468 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2e1f61-2db7-491c-96ef-7af2f856ac10 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:22.112842 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2e1f61-2db7-491c-96ef-7af2f856ac10 None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:22.119984 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 656/1301] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Sep 18 00:47:21 2022] GET /identity/v3/services => generated 2191 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:47:24.313782 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f382f52c-530c-4b99-a5d9-4b9114ab33ce None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:24.314333 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f382f52c-530c-4b99-a5d9-4b9114ab33ce None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:24.314333 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f382f52c-530c-4b99-a5d9-4b9114ab33ce None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:24.315708 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 646/1302] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:47:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:47:24.321744 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab7475d-1bff-4782-b279-4e2f141f7302 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:24.322171 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab7475d-1bff-4782-b279-4e2f141f7302 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:24.322321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab7475d-1bff-4782-b279-4e2f141f7302 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:24.340869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-9ab7475d-1bff-4782-b279-4e2f141f7302 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:47:24.342068 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-9ab7475d-1bff-4782-b279-4e2f141f7302 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:47:24.419631 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 657/1303] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:47:24 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 18 00:47:24.693286 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f382f52c-530c-4b99-a5d9-4b9114ab33ce None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:24.790588 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:24.798210 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f382f52c-530c-4b99-a5d9-4b9114ab33ce None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:24.833334 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea0be3f-f54f-4229-bf12-78b830452571 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:24.833965 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea0be3f-f54f-4229-bf12-78b830452571 None admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:24.834241 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea0be3f-f54f-4229-bf12-78b830452571 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:24.834543 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea0be3f-f54f-4229-bf12-78b830452571 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:24.835315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea0be3f-f54f-4229-bf12-78b830452571 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:24.836697 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea0be3f-f54f-4229-bf12-78b830452571 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:24.847127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application [None req-7ea0be3f-f54f-4229-bf12-78b830452571 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 18 00:47:24.847127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:47:24.847127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 18 00:47:24.847127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 18 00:47:24.847127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 18 00:47:24.847127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 18 00:47:24.847127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 18 00:47:24.847127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:47:24.847127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 18 00:47:24.847127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:47:24.847127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 18 00:47:24.847127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:47:24.847127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 18 00:47:24.847127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 18 00:47:24.847127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 18 00:47:24.847127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 18 00:47:24.847127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 18 00:47:24.848384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 18 00:47:24.848384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 18 00:47:24.848384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 18 00:47:24.848384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 18 00:47:24.848384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 18 00:47:24.848384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 18 00:47:24.848384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 18 00:47:24.848384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 18 00:47:24.848384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 18 00:47:24.848384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 18 00:47:24.848384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 18 00:47:24.848384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 18 00:47:24.848384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 18 00:47:24.848384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 18 00:47:24.848384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 18 00:47:24.848384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.get_or_create( Sep 18 00:47:24.849384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 18 00:47:24.849384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application with Lock( Sep 18 00:47:24.849384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 18 00:47:24.849384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self._enter() Sep 18 00:47:24.849384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 18 00:47:24.849384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 18 00:47:24.849384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 18 00:47:24.849384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application return self.creator() Sep 18 00:47:24.849384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 18 00:47:24.849384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application created_value = creator( Sep 18 00:47:24.849384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 18 00:47:24.849384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 18 00:47:24.849384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 18 00:47:24.849384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: ERROR keystone.server.flask.application Sep 18 00:47:24.849384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 647/1304] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Sep 18 00:47:24 2022] GET /identity/v3/services/placement => generated 90 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 18 00:47:24.854920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab7475d-1bff-4782-b279-4e2f141f7302 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:24.932718 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:24.956895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab7475d-1bff-4782-b279-4e2f141f7302 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:24.990754 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d781c2-5541-42ca-853e-7a8176bb120c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:24.991366 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d781c2-5541-42ca-853e-7a8176bb120c None admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:24.991646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d781c2-5541-42ca-853e-7a8176bb120c None admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:24.991919 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d781c2-5541-42ca-853e-7a8176bb120c None admin] PATH_INFO: `/v3/services` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:24.992557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d781c2-5541-42ca-853e-7a8176bb120c None admin] RBAC: Adding query filter params (name=placement) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:47:24.992886 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d781c2-5541-42ca-853e-7a8176bb120c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:24.994328 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d781c2-5541-42ca-853e-7a8176bb120c None admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:24.999789 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 658/1305] 10.222.0.42 () {56 vars in 1116 bytes} [Sun Sep 18 00:47:24 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:47:25.005797 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea0be3f-f54f-4229-bf12-78b830452571 None admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:47:25.109470 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea0be3f-f54f-4229-bf12-78b830452571 None admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:47:25.144188 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e190856-c903-4506-814f-f2b2dd174526 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:47:25.144795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e190856-c903-4506-814f-f2b2dd174526 None admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:47:25.145071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e190856-c903-4506-814f-f2b2dd174526 None admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:47:25.145341 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e190856-c903-4506-814f-f2b2dd174526 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:47:25.146011 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e190856-c903-4506-814f-f2b2dd174526 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:47:25.147449 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e190856-c903-4506-814f-f2b2dd174526 None admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:47:25.169047 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 648/1306] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:47:25 2022] POST /identity/v3/endpoints => generated 331 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:47:34.996608 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=30 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:47:35.164183 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:48:35.019877 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c19fd8-e339-4261-863d-19863f2b54c5 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:35.020514 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c19fd8-e339-4261-863d-19863f2b54c5 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:35.020665 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c19fd8-e339-4261-863d-19863f2b54c5 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:35.029097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32573c4-ef79-4ce9-8249-a25d32af9742 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:35.029605 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32573c4-ef79-4ce9-8249-a25d32af9742 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:35.029686 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32573c4-ef79-4ce9-8249-a25d32af9742 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:35.035152 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 659/1307] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:48:35 2022] GET /identity => generated 270 bytes in 16 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:48:35.184659 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-d32573c4-ef79-4ce9-8249-a25d32af9742 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:48:35.185945 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d32573c4-ef79-4ce9-8249-a25d32af9742 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:35.330309 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 649/1308] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:48:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 302 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:48:35.648496 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2542beee-90b6-48e2-bef5-7b6c16e3c149 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:35.649699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2542beee-90b6-48e2-bef5-7b6c16e3c149 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:35.650117 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2542beee-90b6-48e2-bef5-7b6c16e3c149 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:35.652046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 660/1309] 10.222.0.42 () {54 vars in 963 bytes} [Sun Sep 18 00:48:35 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:48:35.658091 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8e58b8-c900-455b-bdc2-0cc3e1a656f8 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:35.658390 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8e58b8-c900-455b-bdc2-0cc3e1a656f8 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:35.658741 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8e58b8-c900-455b-bdc2-0cc3e1a656f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:35.764544 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-7f8e58b8-c900-455b-bdc2-0cc3e1a656f8 None None] MFA Rules not processed for user `9b0510812b60454586a781e6399e77a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:48:35.766370 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-7f8e58b8-c900-455b-bdc2-0cc3e1a656f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:35.790285 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:48:35.863647 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 650/1310] 10.222.0.42 () {58 vars in 1060 bytes} [Sun Sep 18 00:48:35 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:48:35.864742 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2542beee-90b6-48e2-bef5-7b6c16e3c149 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:48:35.866666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-2542beee-90b6-48e2-bef5-7b6c16e3c149 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:35.979169 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2542beee-90b6-48e2-bef5-7b6c16e3c149 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:48:36.066419 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84ccb90-f267-44f1-a745-71eccfb84764 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:48:36.067728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84ccb90-f267-44f1-a745-71eccfb84764 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:36.068182 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84ccb90-f267-44f1-a745-71eccfb84764 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:36.068589 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84ccb90-f267-44f1-a745-71eccfb84764 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:36.070363 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-e84ccb90-f267-44f1-a745-71eccfb84764 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:36.140002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:48:36.208173 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84ccb90-f267-44f1-a745-71eccfb84764 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:48:36.210166 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84ccb90-f267-44f1-a745-71eccfb84764 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:48:36.235099 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 661/1311] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:48:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:48:38.839037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27ae5aa-f65f-4ae4-b304-16d88f738251 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:38.839601 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27ae5aa-f65f-4ae4-b304-16d88f738251 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:38.839601 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27ae5aa-f65f-4ae4-b304-16d88f738251 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:38.840978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 651/1312] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:48:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:48:38.846440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3089d2fb-b6dc-4cec-a9a6-898fef1b9d24 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:38.846934 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3089d2fb-b6dc-4cec-a9a6-898fef1b9d24 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:38.847023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3089d2fb-b6dc-4cec-a9a6-898fef1b9d24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:38.920995 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-3089d2fb-b6dc-4cec-a9a6-898fef1b9d24 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:48:38.922214 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3089d2fb-b6dc-4cec-a9a6-898fef1b9d24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:39.020665 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 662/1313] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:48:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:48:39.320761 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27ae5aa-f65f-4ae4-b304-16d88f738251 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:48:39.322480 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-e27ae5aa-f65f-4ae4-b304-16d88f738251 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:39.426935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27ae5aa-f65f-4ae4-b304-16d88f738251 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:48:39.502317 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:48:39.504683 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e42f73e-2f4d-45fa-bf29-a4ddda7ee38c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:48:39.505514 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e42f73e-2f4d-45fa-bf29-a4ddda7ee38c service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:39.505954 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e42f73e-2f4d-45fa-bf29-a4ddda7ee38c service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:39.506273 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e42f73e-2f4d-45fa-bf29-a4ddda7ee38c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:39.508171 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-0e42f73e-2f4d-45fa-bf29-a4ddda7ee38c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:39.618519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e42f73e-2f4d-45fa-bf29-a4ddda7ee38c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:48:39.619888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e42f73e-2f4d-45fa-bf29-a4ddda7ee38c service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:48:39.637579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 652/1314] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:48:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:48:42.126639 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f196afb-835d-4076-a6d1-e1faf671289f None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:42.127221 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f196afb-835d-4076-a6d1-e1faf671289f None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:42.127221 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f196afb-835d-4076-a6d1-e1faf671289f None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:42.128549 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 663/1315] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:48:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:48:42.134678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c65757-f3d4-441b-97e7-4123669a2f87 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:42.135146 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c65757-f3d4-441b-97e7-4123669a2f87 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:42.135248 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c65757-f3d4-441b-97e7-4123669a2f87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:42.204175 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-47c65757-f3d4-441b-97e7-4123669a2f87 None None] MFA Rules not processed for user `796c8ea379cf4e7c9570bc2fdba53211`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:48:42.205426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-47c65757-f3d4-441b-97e7-4123669a2f87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:42.258905 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:48:42.306374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 653/1316] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:48:42 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:48:42.592406 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f196afb-835d-4076-a6d1-e1faf671289f None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:48:42.593937 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-4f196afb-835d-4076-a6d1-e1faf671289f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:42.604421 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:48:42.711562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f196afb-835d-4076-a6d1-e1faf671289f None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:48:42.794405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593a7343-087c-4e8f-b715-cb0855ac8979 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:48:42.794975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593a7343-087c-4e8f-b715-cb0855ac8979 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:42.795254 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593a7343-087c-4e8f-b715-cb0855ac8979 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:42.795527 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593a7343-087c-4e8f-b715-cb0855ac8979 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:42.797144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-593a7343-087c-4e8f-b715-cb0855ac8979 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:42.862965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:48:42.911971 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593a7343-087c-4e8f-b715-cb0855ac8979 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:48:42.913414 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593a7343-087c-4e8f-b715-cb0855ac8979 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:48:42.932457 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 664/1317] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:48:42 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:48:45.728596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b033073-ada4-49fb-9c64-200bde168a04 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:45.729153 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b033073-ada4-49fb-9c64-200bde168a04 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:45.729153 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b033073-ada4-49fb-9c64-200bde168a04 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:45.730543 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 654/1318] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:48:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:48:45.741392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0fa1ec-039d-4b4b-bb1b-831a2deb4f81 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:45.741930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0fa1ec-039d-4b4b-bb1b-831a2deb4f81 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:45.741930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0fa1ec-039d-4b4b-bb1b-831a2deb4f81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:45.817652 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-8d0fa1ec-039d-4b4b-bb1b-831a2deb4f81 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:48:45.818929 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-8d0fa1ec-039d-4b4b-bb1b-831a2deb4f81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:45.922389 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 665/1319] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:48:45 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:48:46.212508 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b033073-ada4-49fb-9c64-200bde168a04 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:48:46.214079 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-1b033073-ada4-49fb-9c64-200bde168a04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:46.318854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b033073-ada4-49fb-9c64-200bde168a04 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:48:46.392415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6831a0e1-c21d-496e-9262-359373137b8a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:48:46.393027 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6831a0e1-c21d-496e-9262-359373137b8a service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:46.393299 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6831a0e1-c21d-496e-9262-359373137b8a service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:46.393577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6831a0e1-c21d-496e-9262-359373137b8a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:46.395315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-6831a0e1-c21d-496e-9262-359373137b8a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:46.403002 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:48:46.510040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6831a0e1-c21d-496e-9262-359373137b8a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:48:46.511420 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6831a0e1-c21d-496e-9262-359373137b8a service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:48:46.531404 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 655/1320] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:48:46 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:48:49.541656 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5116b8a-93a8-4bff-a75c-8896bdfb4f17 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:49.542251 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5116b8a-93a8-4bff-a75c-8896bdfb4f17 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:49.542251 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5116b8a-93a8-4bff-a75c-8896bdfb4f17 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:49.543949 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 666/1321] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:48:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:48:49.549762 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73928709-9dd6-47cc-9b67-4acc1a3ec2f6 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:49.550291 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73928709-9dd6-47cc-9b67-4acc1a3ec2f6 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:49.550381 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73928709-9dd6-47cc-9b67-4acc1a3ec2f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:49.629861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-73928709-9dd6-47cc-9b67-4acc1a3ec2f6 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:48:49.631607 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-73928709-9dd6-47cc-9b67-4acc1a3ec2f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:49.704535 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:48:49.748285 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 656/1322] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:48:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:48:50.060955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3f9b99-cd90-40a4-af25-af8036ca39f9 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:50.061218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3f9b99-cd90-40a4-af25-af8036ca39f9 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:50.061489 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3f9b99-cd90-40a4-af25-af8036ca39f9 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:50.062885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 667/1323] 10.222.0.42 () {54 vars in 963 bytes} [Sun Sep 18 00:48:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:48:50.069148 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820ab5e5-f4e2-484f-a1ab-efc76a15b1fe None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:50.069463 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820ab5e5-f4e2-484f-a1ab-efc76a15b1fe None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:50.069737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820ab5e5-f4e2-484f-a1ab-efc76a15b1fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:50.150248 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-820ab5e5-f4e2-484f-a1ab-efc76a15b1fe None None] MFA Rules not processed for user `9b0510812b60454586a781e6399e77a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:48:50.151647 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-820ab5e5-f4e2-484f-a1ab-efc76a15b1fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:50.230903 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 657/1324] 10.222.0.42 () {58 vars in 1060 bytes} [Sun Sep 18 00:48:50 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:48:50.239878 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3f9b99-cd90-40a4-af25-af8036ca39f9 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:48:50.241406 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-bb3f9b99-cd90-40a4-af25-af8036ca39f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:50.262670 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:48:50.353355 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3f9b99-cd90-40a4-af25-af8036ca39f9 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:48:50.440195 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f06de99-5db0-43b3-9988-0ca20dfb784c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:48:50.440815 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f06de99-5db0-43b3-9988-0ca20dfb784c service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:50.441097 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f06de99-5db0-43b3-9988-0ca20dfb784c service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:50.441373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f06de99-5db0-43b3-9988-0ca20dfb784c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:50.443024 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-1f06de99-5db0-43b3-9988-0ca20dfb784c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:50.529671 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:48:50.569456 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f06de99-5db0-43b3-9988-0ca20dfb784c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:48:50.571454 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f06de99-5db0-43b3-9988-0ca20dfb784c service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:48:50.591299 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 668/1325] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:48:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:48:53.100993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73a7d8c-5e94-4185-98ef-f99772a5d35f None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:53.101559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73a7d8c-5e94-4185-98ef-f99772a5d35f None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:53.101669 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73a7d8c-5e94-4185-98ef-f99772a5d35f None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:53.102988 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 658/1326] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:48:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:48:53.108617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48877c39-f5fd-4311-9a6b-72e9f21bf2d3 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:53.109071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48877c39-f5fd-4311-9a6b-72e9f21bf2d3 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:53.109170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48877c39-f5fd-4311-9a6b-72e9f21bf2d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:53.181511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-48877c39-f5fd-4311-9a6b-72e9f21bf2d3 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:48:53.182732 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-48877c39-f5fd-4311-9a6b-72e9f21bf2d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:53.282389 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 669/1327] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:48:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:48:53.564209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73a7d8c-5e94-4185-98ef-f99772a5d35f None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:48:53.565744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-f73a7d8c-5e94-4185-98ef-f99772a5d35f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:53.629640 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:48:53.673870 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73a7d8c-5e94-4185-98ef-f99772a5d35f None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:48:53.749922 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bb5d16-30ca-415d-93a1-79ea3117ed51 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:48:53.750555 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bb5d16-30ca-415d-93a1-79ea3117ed51 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:53.750837 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bb5d16-30ca-415d-93a1-79ea3117ed51 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:53.751110 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bb5d16-30ca-415d-93a1-79ea3117ed51 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:53.752702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-69bb5d16-30ca-415d-93a1-79ea3117ed51 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:53.859651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:48:53.862357 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bb5d16-30ca-415d-93a1-79ea3117ed51 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:48:53.864131 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bb5d16-30ca-415d-93a1-79ea3117ed51 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:48:53.883173 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 659/1328] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:48:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:48:56.114874 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4432b89-3dc8-4bc4-9b77-3cb6adb59818 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:56.115389 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4432b89-3dc8-4bc4-9b77-3cb6adb59818 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:56.115478 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4432b89-3dc8-4bc4-9b77-3cb6adb59818 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:56.116823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 670/1329] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:48:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:48:56.122534 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb50ade-93e5-45e8-a0bf-f046a4e66b4c None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:56.122978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb50ade-93e5-45e8-a0bf-f046a4e66b4c None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:56.123084 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb50ade-93e5-45e8-a0bf-f046a4e66b4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:56.192049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-2eb50ade-93e5-45e8-a0bf-f046a4e66b4c None None] MFA Rules not processed for user `796c8ea379cf4e7c9570bc2fdba53211`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:48:56.193274 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-2eb50ade-93e5-45e8-a0bf-f046a4e66b4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:56.297905 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 660/1330] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:48:56 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:48:56.584342 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4432b89-3dc8-4bc4-9b77-3cb6adb59818 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:48:56.586484 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a4432b89-3dc8-4bc4-9b77-3cb6adb59818 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:56.617434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:48:56.696390 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4432b89-3dc8-4bc4-9b77-3cb6adb59818 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:48:56.772971 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b3f24b-7efc-4fc5-9240-c4b9b17e700b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:48:56.773588 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b3f24b-7efc-4fc5-9240-c4b9b17e700b service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:56.773862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b3f24b-7efc-4fc5-9240-c4b9b17e700b service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:56.774131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b3f24b-7efc-4fc5-9240-c4b9b17e700b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:56.775743 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-26b3f24b-7efc-4fc5-9240-c4b9b17e700b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:56.854776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:48:56.889298 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b3f24b-7efc-4fc5-9240-c4b9b17e700b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:48:56.890773 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b3f24b-7efc-4fc5-9240-c4b9b17e700b service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:48:56.909000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 671/1331] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:48:56 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:48:59.344599 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63dc794-a333-4321-8690-ce6ee176c85c None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:59.345238 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63dc794-a333-4321-8690-ce6ee176c85c None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:59.345238 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63dc794-a333-4321-8690-ce6ee176c85c None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:59.346508 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 661/1332] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:48:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:48:59.352002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1202b215-dc60-4bb4-91c7-ca7319eff2f3 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:59.352440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1202b215-dc60-4bb4-91c7-ca7319eff2f3 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:59.352530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1202b215-dc60-4bb4-91c7-ca7319eff2f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:59.419632 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-1202b215-dc60-4bb4-91c7-ca7319eff2f3 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:48:59.420849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-1202b215-dc60-4bb4-91c7-ca7319eff2f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:59.519404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 672/1333] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:48:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:48:59.803158 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63dc794-a333-4321-8690-ce6ee176c85c None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:48:59.805192 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d63dc794-a333-4321-8690-ce6ee176c85c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:48:59.874523 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:48:59.908966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63dc794-a333-4321-8690-ce6ee176c85c None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:48:59.984052 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caeda917-4060-41e4-87f5-99ea98b185f9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:48:59.984663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caeda917-4060-41e4-87f5-99ea98b185f9 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:48:59.984937 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caeda917-4060-41e4-87f5-99ea98b185f9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:48:59.985203 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caeda917-4060-41e4-87f5-99ea98b185f9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:48:59.986893 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-caeda917-4060-41e4-87f5-99ea98b185f9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:00.098725 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caeda917-4060-41e4-87f5-99ea98b185f9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:49:00.100173 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caeda917-4060-41e4-87f5-99ea98b185f9 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:49:00.110354 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:00.119093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 662/1334] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:48:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:49:02.357896 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177ac3da-c4b1-413a-8f0f-ed1eda886c9c None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:02.358476 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177ac3da-c4b1-413a-8f0f-ed1eda886c9c None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:02.358476 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177ac3da-c4b1-413a-8f0f-ed1eda886c9c None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:02.359847 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 673/1335] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:49:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:49:02.365220 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6357eaf-441a-487b-8d8b-a9383310ad7d None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:02.365601 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6357eaf-441a-487b-8d8b-a9383310ad7d None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:02.365780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6357eaf-441a-487b-8d8b-a9383310ad7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:02.435654 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-c6357eaf-441a-487b-8d8b-a9383310ad7d None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:49:02.436791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c6357eaf-441a-487b-8d8b-a9383310ad7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:02.541080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 663/1336] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:49:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:49:02.824674 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177ac3da-c4b1-413a-8f0f-ed1eda886c9c None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:49:02.826208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-177ac3da-c4b1-413a-8f0f-ed1eda886c9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:02.870592 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:02.935417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177ac3da-c4b1-413a-8f0f-ed1eda886c9c None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:49:03.011638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7ece41-6081-4c58-bf27-d68e9b4ab121 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:49:03.012252 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7ece41-6081-4c58-bf27-d68e9b4ab121 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:03.012529 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7ece41-6081-4c58-bf27-d68e9b4ab121 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:03.012802 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7ece41-6081-4c58-bf27-d68e9b4ab121 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:03.014416 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-2c7ece41-6081-4c58-bf27-d68e9b4ab121 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:03.103164 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:03.131216 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7ece41-6081-4c58-bf27-d68e9b4ab121 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:49:03.132779 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7ece41-6081-4c58-bf27-d68e9b4ab121 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:49:03.151592 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 674/1337] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:49:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:49:05.720865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61479715-6449-4ec4-be9c-ea5f5cf17beb None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:05.721407 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61479715-6449-4ec4-be9c-ea5f5cf17beb None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:05.721407 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61479715-6449-4ec4-be9c-ea5f5cf17beb None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:05.722784 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 664/1338] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:49:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:49:05.729061 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4908f43-5074-4193-8a9b-6185bd63fa90 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:05.729556 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4908f43-5074-4193-8a9b-6185bd63fa90 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:05.729658 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4908f43-5074-4193-8a9b-6185bd63fa90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:05.797852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-f4908f43-5074-4193-8a9b-6185bd63fa90 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:49:05.799019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-f4908f43-5074-4193-8a9b-6185bd63fa90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:05.906932 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 675/1339] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:49:05 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:49:06.194316 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61479715-6449-4ec4-be9c-ea5f5cf17beb None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:49:06.202824 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-61479715-6449-4ec4-be9c-ea5f5cf17beb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:06.298093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:06.346613 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61479715-6449-4ec4-be9c-ea5f5cf17beb None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:49:06.430147 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af889246-4705-4e85-ab27-22f481c31771 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:49:06.430708 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af889246-4705-4e85-ab27-22f481c31771 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:06.430988 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af889246-4705-4e85-ab27-22f481c31771 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:06.431257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af889246-4705-4e85-ab27-22f481c31771 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:06.432948 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-af889246-4705-4e85-ab27-22f481c31771 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:06.548986 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af889246-4705-4e85-ab27-22f481c31771 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:49:06.550420 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af889246-4705-4e85-ab27-22f481c31771 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:49:06.569805 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 665/1340] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:49:06 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:49:10.844157 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10213d2a-90fe-4e62-b85b-dcf4cc10d8a8 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:10.845209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10213d2a-90fe-4e62-b85b-dcf4cc10d8a8 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:10.845209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10213d2a-90fe-4e62-b85b-dcf4cc10d8a8 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:10.846047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 676/1341] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:49:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:49:10.852190 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51bd16c-61da-487e-a17f-d26958909a8f None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:10.852700 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51bd16c-61da-487e-a17f-d26958909a8f None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:10.852798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51bd16c-61da-487e-a17f-d26958909a8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:10.858493 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:10.932391 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-d51bd16c-61da-487e-a17f-d26958909a8f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:49:10.933642 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d51bd16c-61da-487e-a17f-d26958909a8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:11.055517 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 666/1342] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:49:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:49:11.346842 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10213d2a-90fe-4e62-b85b-dcf4cc10d8a8 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:49:11.348691 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-10213d2a-90fe-4e62-b85b-dcf4cc10d8a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:11.405403 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:11.477144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10213d2a-90fe-4e62-b85b-dcf4cc10d8a8 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:49:11.605519 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20aa710-c865-4971-b30b-d3222a4480a0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:49:11.606431 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20aa710-c865-4971-b30b-d3222a4480a0 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:11.606944 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20aa710-c865-4971-b30b-d3222a4480a0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:11.607415 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20aa710-c865-4971-b30b-d3222a4480a0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:11.611077 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-e20aa710-c865-4971-b30b-d3222a4480a0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:11.736047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:11.754683 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20aa710-c865-4971-b30b-d3222a4480a0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:49:11.756142 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20aa710-c865-4971-b30b-d3222a4480a0 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:49:11.776275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 677/1343] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:49:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:49:15.053244 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebd8255-9d99-4425-a517-fedd99d77326 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:15.053811 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebd8255-9d99-4425-a517-fedd99d77326 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:15.053811 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebd8255-9d99-4425-a517-fedd99d77326 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:15.057926 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 667/1344] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:49:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:49:15.062883 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1dd017-dd8c-4709-90c2-850261b6a081 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:15.063429 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1dd017-dd8c-4709-90c2-850261b6a081 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:15.063566 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1dd017-dd8c-4709-90c2-850261b6a081 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:15.150033 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-1c1dd017-dd8c-4709-90c2-850261b6a081 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:49:15.151252 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-1c1dd017-dd8c-4709-90c2-850261b6a081 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:15.266991 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 678/1345] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:49:15 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:49:15.580251 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ebd8255-9d99-4425-a517-fedd99d77326 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:49:15.581777 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-6ebd8255-9d99-4425-a517-fedd99d77326 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:15.694368 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:15.713957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ebd8255-9d99-4425-a517-fedd99d77326 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:49:15.807184 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6f2018-3cea-4379-8e2a-ad5dd3f61907 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:49:15.807805 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6f2018-3cea-4379-8e2a-ad5dd3f61907 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:15.808084 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6f2018-3cea-4379-8e2a-ad5dd3f61907 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:15.808358 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6f2018-3cea-4379-8e2a-ad5dd3f61907 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:15.810275 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-5e6f2018-3cea-4379-8e2a-ad5dd3f61907 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:15.952841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6f2018-3cea-4379-8e2a-ad5dd3f61907 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:49:15.954680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6f2018-3cea-4379-8e2a-ad5dd3f61907 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:49:15.979553 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 668/1346] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:49:15 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:49:20.670326 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9697cea5-59c2-4d2e-8f4e-fafde4592011 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:20.670965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9697cea5-59c2-4d2e-8f4e-fafde4592011 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:20.670965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9697cea5-59c2-4d2e-8f4e-fafde4592011 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:20.672490 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 679/1347] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:49:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:49:20.679188 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde6fd42-cfc1-4fb4-96c0-9f31b04a65d2 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:20.679870 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde6fd42-cfc1-4fb4-96c0-9f31b04a65d2 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:20.679870 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde6fd42-cfc1-4fb4-96c0-9f31b04a65d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:20.702560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:20.771834 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-fde6fd42-cfc1-4fb4-96c0-9f31b04a65d2 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:49:20.773650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-fde6fd42-cfc1-4fb4-96c0-9f31b04a65d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:20.908410 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 669/1348] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:49:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:49:21.208232 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9697cea5-59c2-4d2e-8f4e-fafde4592011 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:49:21.209821 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-9697cea5-59c2-4d2e-8f4e-fafde4592011 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:21.270857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:21.330241 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9697cea5-59c2-4d2e-8f4e-fafde4592011 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:49:21.417702 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7378f790-3aab-4a03-9101-1766e6fb6d70 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:49:21.418334 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7378f790-3aab-4a03-9101-1766e6fb6d70 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:21.418653 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7378f790-3aab-4a03-9101-1766e6fb6d70 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:21.418929 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7378f790-3aab-4a03-9101-1766e6fb6d70 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:21.420712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-7378f790-3aab-4a03-9101-1766e6fb6d70 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:21.585015 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:21.591762 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7378f790-3aab-4a03-9101-1766e6fb6d70 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:49:21.594796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7378f790-3aab-4a03-9101-1766e6fb6d70 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:49:21.628849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 680/1349] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:49:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:49:28.024499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6468b755-807a-433c-abe5-383c3ff55977 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:28.025017 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6468b755-807a-433c-abe5-383c3ff55977 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:28.025017 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6468b755-807a-433c-abe5-383c3ff55977 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:28.026968 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 670/1350] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 18 00:49:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:49:28.032862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5173d45b-c4c5-4bdf-985f-3d1f612b73ad None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:28.033385 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5173d45b-c4c5-4bdf-985f-3d1f612b73ad None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:28.033492 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5173d45b-c4c5-4bdf-985f-3d1f612b73ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:28.138121 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-5173d45b-c4c5-4bdf-985f-3d1f612b73ad None None] MFA Rules not processed for user `8f30116fb5ce43dab69efa8bafd9d049`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:49:28.139934 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-5173d45b-c4c5-4bdf-985f-3d1f612b73ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:28.327726 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 681/1351] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 18 00:49:28 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 296 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:49:30.906680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=59 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:32.328385 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d40dad-e4e8-479b-a27a-5bc7a292c970 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:32.329293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d40dad-e4e8-479b-a27a-5bc7a292c970 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:32.329293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d40dad-e4e8-479b-a27a-5bc7a292c970 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:32.330642 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 671/1352] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 18 00:49:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:49:32.337481 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c91232-f8a4-4a46-bd39-e93e04b5483b None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:32.338964 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c91232-f8a4-4a46-bd39-e93e04b5483b None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:32.339788 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c91232-f8a4-4a46-bd39-e93e04b5483b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:32.457757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:32.483582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-83c91232-f8a4-4a46-bd39-e93e04b5483b None None] MFA Rules not processed for user `8f30116fb5ce43dab69efa8bafd9d049`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:49:32.485343 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-83c91232-f8a4-4a46-bd39-e93e04b5483b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:32.660667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 682/1353] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 18 00:49:32 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 325 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:49:34.418880 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c06011e-0749-4a1d-b0b2-cf9c58c11439 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:34.419412 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c06011e-0749-4a1d-b0b2-cf9c58c11439 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:34.419972 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c06011e-0749-4a1d-b0b2-cf9c58c11439 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:34.422391 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 672/1354] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 18 00:49:34 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:49:34.430335 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7002842-9c67-4a6d-bcc0-c97144bbfa32 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:34.431271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7002842-9c67-4a6d-bcc0-c97144bbfa32 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:34.431271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7002842-9c67-4a6d-bcc0-c97144bbfa32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:34.528522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-c7002842-9c67-4a6d-bcc0-c97144bbfa32 None None] MFA Rules not processed for user `8f30116fb5ce43dab69efa8bafd9d049`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:49:34.529893 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c7002842-9c67-4a6d-bcc0-c97144bbfa32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:34.581742 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:34.664698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 683/1355] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 18 00:49:34 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:49:38.159835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb4731f-7583-47a7-8e57-6dc0574cbdfa None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:38.160500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb4731f-7583-47a7-8e57-6dc0574cbdfa None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:38.160500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb4731f-7583-47a7-8e57-6dc0574cbdfa None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:38.162039 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 673/1356] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:49:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:49:38.167872 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab509621-73fd-4c97-bfa8-a9e425176a02 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:38.170008 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab509621-73fd-4c97-bfa8-a9e425176a02 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:38.170008 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab509621-73fd-4c97-bfa8-a9e425176a02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:38.317678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-ab509621-73fd-4c97-bfa8-a9e425176a02 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:49:38.320247 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-ab509621-73fd-4c97-bfa8-a9e425176a02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:38.485024 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 684/1357] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:49:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:49:38.790967 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7dfbb3-4fcf-484f-a065-0e74f88b38ca None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:38.793034 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7dfbb3-4fcf-484f-a065-0e74f88b38ca None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:38.793485 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7dfbb3-4fcf-484f-a065-0e74f88b38ca None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:38.795521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 674/1358] 10.222.0.42 () {54 vars in 965 bytes} [Sun Sep 18 00:49:38 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:49:38.805369 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c88e6a-58ae-4a44-9551-45f2a2e81b0e None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:38.806326 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c88e6a-58ae-4a44-9551-45f2a2e81b0e None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:38.807649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c88e6a-58ae-4a44-9551-45f2a2e81b0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:38.832083 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:38.903088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-64c88e6a-58ae-4a44-9551-45f2a2e81b0e None None] MFA Rules not processed for user `a59d73fd45a54651930d52300704fdb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:49:38.904386 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-64c88e6a-58ae-4a44-9551-45f2a2e81b0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:39.086340 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 685/1359] 10.222.0.42 () {58 vars in 1062 bytes} [Sun Sep 18 00:49:38 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 283 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:49:39.099411 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7dfbb3-4fcf-484f-a065-0e74f88b38ca None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:49:39.101031 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-3f7dfbb3-4fcf-484f-a065-0e74f88b38ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:39.272008 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7dfbb3-4fcf-484f-a065-0e74f88b38ca None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:49:39.371002 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:39.421304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd06d232-d515-480f-8677-6b8a63208f34 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:49:39.421939 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd06d232-d515-480f-8677-6b8a63208f34 service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:39.422311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd06d232-d515-480f-8677-6b8a63208f34 service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:39.422609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd06d232-d515-480f-8677-6b8a63208f34 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:39.424588 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-dd06d232-d515-480f-8677-6b8a63208f34 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:39.566219 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd06d232-d515-480f-8677-6b8a63208f34 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:49:39.569103 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd06d232-d515-480f-8677-6b8a63208f34 service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:49:39.607011 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 675/1360] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:49:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 508 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:49:42.457068 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27aaf559-18ea-4c58-b76c-7541f23e3137 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:42.457955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27aaf559-18ea-4c58-b76c-7541f23e3137 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:42.457955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27aaf559-18ea-4c58-b76c-7541f23e3137 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:42.459086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 686/1361] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:49:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:49:42.467726 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c708197-0811-4342-9fc5-295b87586810 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:42.468456 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c708197-0811-4342-9fc5-295b87586810 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:42.469081 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c708197-0811-4342-9fc5-295b87586810 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:42.592626 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-4c708197-0811-4342-9fc5-295b87586810 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:49:42.594772 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-4c708197-0811-4342-9fc5-295b87586810 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:42.671690 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:42.790742 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 676/1362] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:49:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 325 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:49:43.342139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27aaf559-18ea-4c58-b76c-7541f23e3137 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:49:43.343688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-27aaf559-18ea-4c58-b76c-7541f23e3137 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:43.426710 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:43.528927 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27aaf559-18ea-4c58-b76c-7541f23e3137 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:49:43.661567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e04cc51-602a-4985-821f-6c9dd83d39f7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:49:43.662280 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e04cc51-602a-4985-821f-6c9dd83d39f7 service nova] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:43.662660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e04cc51-602a-4985-821f-6c9dd83d39f7 service nova] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:43.663516 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e04cc51-602a-4985-821f-6c9dd83d39f7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:43.665229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-7e04cc51-602a-4985-821f-6c9dd83d39f7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:43.691703 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:43.765354 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e04cc51-602a-4985-821f-6c9dd83d39f7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:49:43.767122 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e04cc51-602a-4985-821f-6c9dd83d39f7 service nova] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:49:43.786886 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 687/1363] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:49:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:49:46.389334 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1ff56b-b560-497a-b85d-9ff5b644b753 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:46.390177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1ff56b-b560-497a-b85d-9ff5b644b753 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:46.390177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1ff56b-b560-497a-b85d-9ff5b644b753 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:46.391529 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 677/1364] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:49:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:49:46.397857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9e4d14-d877-46b9-b77f-e6b92bc012ba None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:46.398395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9e4d14-d877-46b9-b77f-e6b92bc012ba None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:46.398395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9e4d14-d877-46b9-b77f-e6b92bc012ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:46.474430 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-ef9e4d14-d877-46b9-b77f-e6b92bc012ba None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:49:46.476200 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-ef9e4d14-d877-46b9-b77f-e6b92bc012ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:46.593385 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:46.596161 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 688/1365] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:49:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:49:46.908097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a0eeac-4779-4f52-9dab-9f22ac9e4fad None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:46.908593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a0eeac-4779-4f52-9dab-9f22ac9e4fad None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:46.909115 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a0eeac-4779-4f52-9dab-9f22ac9e4fad None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:46.913194 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 678/1366] 10.222.0.42 () {54 vars in 965 bytes} [Sun Sep 18 00:49:46 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:49:46.923147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc889eda-38da-4747-9b43-28984f6c63c6 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:46.923424 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc889eda-38da-4747-9b43-28984f6c63c6 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:46.923721 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc889eda-38da-4747-9b43-28984f6c63c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:47.031099 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-dc889eda-38da-4747-9b43-28984f6c63c6 None None] MFA Rules not processed for user `a59d73fd45a54651930d52300704fdb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:49:47.032166 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-dc889eda-38da-4747-9b43-28984f6c63c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:47.161315 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 689/1367] 10.222.0.42 () {58 vars in 1062 bytes} [Sun Sep 18 00:49:46 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:49:47.170362 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a0eeac-4779-4f52-9dab-9f22ac9e4fad None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:49:47.173416 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-10a0eeac-4779-4f52-9dab-9f22ac9e4fad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:47.370610 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b682863-85c7-4985-bf41-97a201632291 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:47.370904 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b682863-85c7-4985-bf41-97a201632291 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:47.371186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b682863-85c7-4985-bf41-97a201632291 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:47.372603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 690/1368] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:49:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:49:47.378517 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd929dea-94a9-4a1f-8b89-2fc8958610c8 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:47.378752 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd929dea-94a9-4a1f-8b89-2fc8958610c8 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:47.378917 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd929dea-94a9-4a1f-8b89-2fc8958610c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:47.422019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a0eeac-4779-4f52-9dab-9f22ac9e4fad None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:49:47.452695 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:47.462006 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:47.472453 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-bd929dea-94a9-4a1f-8b89-2fc8958610c8 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:49:47.473698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-bd929dea-94a9-4a1f-8b89-2fc8958610c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:47.587658 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f598d8-6f88-42de-9983-7bd68837ca1b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:49:47.589336 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f598d8-6f88-42de-9983-7bd68837ca1b service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:47.589767 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f598d8-6f88-42de-9983-7bd68837ca1b service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:47.590243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f598d8-6f88-42de-9983-7bd68837ca1b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:47.592518 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-a3f598d8-6f88-42de-9983-7bd68837ca1b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:47.656779 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 691/1369] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:49:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 279 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:49:47.713060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f598d8-6f88-42de-9983-7bd68837ca1b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:49:47.715342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f598d8-6f88-42de-9983-7bd68837ca1b service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:49:47.740486 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 679/1370] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:49:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 571 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:49:48.006218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c11919e-298e-4284-904b-8751d757a865 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:48.006218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c11919e-298e-4284-904b-8751d757a865 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:48.006218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c11919e-298e-4284-904b-8751d757a865 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:48.009635 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 692/1371] 10.222.0.42 () {54 vars in 964 bytes} [Sun Sep 18 00:49:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:49:48.015435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed5b402-8453-4db2-8240-a80361f2984b None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:48.015435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed5b402-8453-4db2-8240-a80361f2984b None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:48.015435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed5b402-8453-4db2-8240-a80361f2984b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:48.056105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:48.133717 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-9ed5b402-8453-4db2-8240-a80361f2984b None None] MFA Rules not processed for user `c4c3eaee0a8f4de0b79841a95b35091f`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:49:48.135008 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-9ed5b402-8453-4db2-8240-a80361f2984b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:48.239243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 680/1372] 10.222.0.42 () {58 vars in 1061 bytes} [Sun Sep 18 00:49:48 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:49:48.257861 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c11919e-298e-4284-904b-8751d757a865 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:49:48.261685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-1c11919e-298e-4284-904b-8751d757a865 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:48.539115 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c11919e-298e-4284-904b-8751d757a865 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:49:48.587757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:48.643329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554ab412-f764-44d7-aff0-2e8b35d5a10f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:49:48.643979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554ab412-f764-44d7-aff0-2e8b35d5a10f service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:48.644260 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554ab412-f764-44d7-aff0-2e8b35d5a10f service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:48.644541 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554ab412-f764-44d7-aff0-2e8b35d5a10f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:48.646154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-554ab412-f764-44d7-aff0-2e8b35d5a10f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:48.767941 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554ab412-f764-44d7-aff0-2e8b35d5a10f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:49:48.769337 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554ab412-f764-44d7-aff0-2e8b35d5a10f service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:49:48.789513 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 693/1373] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:49:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 534 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:49:50.315491 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ac8f92-96cc-481f-83a8-0d6466efe6eb None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:50.316111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ac8f92-96cc-481f-83a8-0d6466efe6eb None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:50.316111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ac8f92-96cc-481f-83a8-0d6466efe6eb None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:50.317823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 681/1374] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:49:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:49:50.325644 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b35ae5-bd83-4277-9e79-ea57ea9859f1 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:50.327230 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b35ae5-bd83-4277-9e79-ea57ea9859f1 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:50.327230 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b35ae5-bd83-4277-9e79-ea57ea9859f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:50.416662 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-11b35ae5-bd83-4277-9e79-ea57ea9859f1 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:49:50.418132 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-11b35ae5-bd83-4277-9e79-ea57ea9859f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:50.426522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:50.556022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 694/1375] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:49:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:49:50.853412 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ac8f92-96cc-481f-83a8-0d6466efe6eb None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:49:50.855032 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c8ac8f92-96cc-481f-83a8-0d6466efe6eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:50.999131 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:51.045663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ac8f92-96cc-481f-83a8-0d6466efe6eb None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:49:51.200021 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdbbc9f-a624-460d-8c81-72643fffdf9f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:49:51.200753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdbbc9f-a624-460d-8c81-72643fffdf9f service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:51.200904 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdbbc9f-a624-460d-8c81-72643fffdf9f service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:51.201182 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdbbc9f-a624-460d-8c81-72643fffdf9f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:51.202911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ffdbbc9f-a624-460d-8c81-72643fffdf9f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:51.318391 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:51.336635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffdbbc9f-a624-460d-8c81-72643fffdf9f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:49:51.338339 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffdbbc9f-a624-460d-8c81-72643fffdf9f service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:49:51.359613 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 682/1376] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:49:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 507 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:49:54.063105 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08f5926-4049-451e-805a-db3bbd51ffdf None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:54.063814 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08f5926-4049-451e-805a-db3bbd51ffdf None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:54.063814 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08f5926-4049-451e-805a-db3bbd51ffdf None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:54.065219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 695/1377] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:49:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:49:54.071960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0ad363-2412-4310-869a-87cc9b68bacb None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:54.072533 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0ad363-2412-4310-869a-87cc9b68bacb None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:54.072533 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0ad363-2412-4310-869a-87cc9b68bacb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:54.155394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-5a0ad363-2412-4310-869a-87cc9b68bacb None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:49:54.156914 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-5a0ad363-2412-4310-869a-87cc9b68bacb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:54.301657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 683/1378] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:49:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:49:54.646768 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08f5926-4049-451e-805a-db3bbd51ffdf None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:49:54.648304 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c08f5926-4049-451e-805a-db3bbd51ffdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:54.809650 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:54.824164 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08f5926-4049-451e-805a-db3bbd51ffdf None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:49:54.976130 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d574b5f3-c95d-4461-8a0d-a9dfa9929cb2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:49:54.976747 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d574b5f3-c95d-4461-8a0d-a9dfa9929cb2 service nova] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:54.977031 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d574b5f3-c95d-4461-8a0d-a9dfa9929cb2 service nova] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:54.977312 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d574b5f3-c95d-4461-8a0d-a9dfa9929cb2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:54.978982 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-d574b5f3-c95d-4461-8a0d-a9dfa9929cb2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:55.107239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d574b5f3-c95d-4461-8a0d-a9dfa9929cb2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:49:55.109347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d574b5f3-c95d-4461-8a0d-a9dfa9929cb2 service nova] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:49:55.126548 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:55.142443 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 696/1379] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:49:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 496 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:49:57.592520 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c05e082-4753-4aa4-82af-ede1d791a19f None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:57.593584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c05e082-4753-4aa4-82af-ede1d791a19f None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:57.593584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c05e082-4753-4aa4-82af-ede1d791a19f None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:57.595449 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 684/1380] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:49:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:49:57.602580 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2555150-1045-4420-bd3c-72d2a5da41b8 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:57.603493 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2555150-1045-4420-bd3c-72d2a5da41b8 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:57.603493 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2555150-1045-4420-bd3c-72d2a5da41b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:57.693344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-b2555150-1045-4420-bd3c-72d2a5da41b8 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:49:57.694646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-b2555150-1045-4420-bd3c-72d2a5da41b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:57.811758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 697/1381] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:49:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:49:58.127793 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c05e082-4753-4aa4-82af-ede1d791a19f None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:49:58.129325 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-5c05e082-4753-4aa4-82af-ede1d791a19f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:58.198478 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:58.299537 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c05e082-4753-4aa4-82af-ede1d791a19f None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:49:58.428448 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25997a1b-389b-4a03-9e74-50fb316643fe service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:49:58.428970 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25997a1b-389b-4a03-9e74-50fb316643fe service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:49:58.429247 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25997a1b-389b-4a03-9e74-50fb316643fe service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:49:58.429512 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25997a1b-389b-4a03-9e74-50fb316643fe service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:49:58.431212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-25997a1b-389b-4a03-9e74-50fb316643fe service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:49:58.452398 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:49:58.541194 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25997a1b-389b-4a03-9e74-50fb316643fe service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:49:58.543397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25997a1b-389b-4a03-9e74-50fb316643fe service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:49:58.571044 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 685/1382] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:49:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 444 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:50:00.893077 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b362ec9f-fe73-43b8-95b7-cf5a9be1c5c4 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:00.893617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b362ec9f-fe73-43b8-95b7-cf5a9be1c5c4 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:00.893806 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b362ec9f-fe73-43b8-95b7-cf5a9be1c5c4 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:00.895703 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 698/1383] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:50:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:50:00.901683 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a820aca-251c-4bd1-bb21-068c27b0bb46 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:00.902133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a820aca-251c-4bd1-bb21-068c27b0bb46 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:00.902265 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a820aca-251c-4bd1-bb21-068c27b0bb46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:00.981204 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-7a820aca-251c-4bd1-bb21-068c27b0bb46 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:00.982489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-7a820aca-251c-4bd1-bb21-068c27b0bb46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:01.116202 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:01.124899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 686/1384] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:50:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:50:01.445215 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b362ec9f-fe73-43b8-95b7-cf5a9be1c5c4 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:50:01.446785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-b362ec9f-fe73-43b8-95b7-cf5a9be1c5c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:01.547132 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:01.636507 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b362ec9f-fe73-43b8-95b7-cf5a9be1c5c4 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:50:01.770363 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddaca491-9d16-4638-b62a-0093f04270f7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:50:01.772298 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaca491-9d16-4638-b62a-0093f04270f7 service nova] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:01.773094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaca491-9d16-4638-b62a-0093f04270f7 service nova] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:01.774851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaca491-9d16-4638-b62a-0093f04270f7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:01.779272 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-ddaca491-9d16-4638-b62a-0093f04270f7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:01.855648 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:01.925178 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddaca491-9d16-4638-b62a-0093f04270f7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:50:01.927375 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddaca491-9d16-4638-b62a-0093f04270f7 service nova] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:50:01.954894 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 699/1385] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:50:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 510 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:50:04.235398 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c160f81-8a7b-4bc3-ab8d-dcc617d6c7f7 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:04.236090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c160f81-8a7b-4bc3-ab8d-dcc617d6c7f7 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:04.236090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c160f81-8a7b-4bc3-ab8d-dcc617d6c7f7 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:04.237306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 687/1386] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:50:04 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:50:04.246089 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15016c79-9cf6-4ddd-8cad-97f5deaddcec None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:04.246889 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15016c79-9cf6-4ddd-8cad-97f5deaddcec None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:04.246889 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15016c79-9cf6-4ddd-8cad-97f5deaddcec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:04.320479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-15016c79-9cf6-4ddd-8cad-97f5deaddcec None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:04.321713 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-15016c79-9cf6-4ddd-8cad-97f5deaddcec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:04.502435 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 700/1387] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:50:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:50:04.794753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c160f81-8a7b-4bc3-ab8d-dcc617d6c7f7 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:50:04.796642 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-6c160f81-8a7b-4bc3-ab8d-dcc617d6c7f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:04.977859 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c160f81-8a7b-4bc3-ab8d-dcc617d6c7f7 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:50:05.048207 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4ce7f6-b2eb-4a74-a42c-bbf457538712 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:05.048436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4ce7f6-b2eb-4a74-a42c-bbf457538712 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:05.048645 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4ce7f6-b2eb-4a74-a42c-bbf457538712 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:05.049994 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 701/1388] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:50:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:50:05.055365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189c5e4a-9f50-411b-9de0-8cfc9e98d1fb None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:05.055706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189c5e4a-9f50-411b-9de0-8cfc9e98d1fb None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:05.055916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189c5e4a-9f50-411b-9de0-8cfc9e98d1fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:05.061258 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:05.074932 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:05.119040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b7f112-9d60-46e0-b1c1-c56b955b9347 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:50:05.119644 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b7f112-9d60-46e0-b1c1-c56b955b9347 service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:05.119918 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b7f112-9d60-46e0-b1c1-c56b955b9347 service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:05.120183 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b7f112-9d60-46e0-b1c1-c56b955b9347 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:05.121802 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c3b7f112-9d60-46e0-b1c1-c56b955b9347 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:05.142447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-189c5e4a-9f50-411b-9de0-8cfc9e98d1fb None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:05.144039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-189c5e4a-9f50-411b-9de0-8cfc9e98d1fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:05.249694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b7f112-9d60-46e0-b1c1-c56b955b9347 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:50:05.252426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b7f112-9d60-46e0-b1c1-c56b955b9347 service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:50:05.275823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 688/1389] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:50:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 482 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:50:05.284522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 702/1390] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:50:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:50:05.596708 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e96f8e-01cf-46e4-b700-eee60c14e1b7 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:05.596980 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e96f8e-01cf-46e4-b700-eee60c14e1b7 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:05.597455 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e96f8e-01cf-46e4-b700-eee60c14e1b7 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:05.600147 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 689/1391] 10.222.0.42 () {54 vars in 962 bytes} [Sun Sep 18 00:50:05 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:50:05.608594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd528bc9-e62a-4a05-ba44-0c63aa6514ac None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:05.609046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd528bc9-e62a-4a05-ba44-0c63aa6514ac None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:05.609565 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd528bc9-e62a-4a05-ba44-0c63aa6514ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:05.719011 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-bd528bc9-e62a-4a05-ba44-0c63aa6514ac None None] MFA Rules not processed for user `d7fe9d21099e4c4891c28687bb673214`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:05.720255 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-bd528bc9-e62a-4a05-ba44-0c63aa6514ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:05.761846 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:05.828158 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 703/1392] 10.222.0.42 () {58 vars in 1059 bytes} [Sun Sep 18 00:50:05 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:50:05.838112 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e96f8e-01cf-46e4-b700-eee60c14e1b7 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:50:05.839747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-79e96f8e-01cf-46e4-b700-eee60c14e1b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:05.945045 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:05.980055 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e96f8e-01cf-46e4-b700-eee60c14e1b7 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:50:06.075650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532e198d-a6c7-4a06-85e2-ae3b4c315acf service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7fe9d21099e4c4891c28687bb673214', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:50:06.076265 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532e198d-a6c7-4a06-85e2-ae3b4c315acf service glance] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:06.076542 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532e198d-a6c7-4a06-85e2-ae3b4c315acf service glance] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:06.076813 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532e198d-a6c7-4a06-85e2-ae3b4c315acf service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:06.080754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-532e198d-a6c7-4a06-85e2-ae3b4c315acf service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:06.226979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-532e198d-a6c7-4a06-85e2-ae3b4c315acf service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:50:06.228426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-532e198d-a6c7-4a06-85e2-ae3b4c315acf service glance] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:50:06.266449 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 690/1393] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:50:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:50:07.132139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b252e5d-d2be-4852-a696-60c6672c1d1a None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:07.132953 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b252e5d-d2be-4852-a696-60c6672c1d1a None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:07.133326 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b252e5d-d2be-4852-a696-60c6672c1d1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:07.215101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-2b252e5d-d2be-4852-a696-60c6672c1d1a None None] MFA Rules not processed for user `3e31dce69c344c9aa4a55f31b9c735df`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:07.216314 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-2b252e5d-d2be-4852-a696-60c6672c1d1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:07.326275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 704/1394] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 18 00:50:07 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:50:07.383471 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7339486f-0fd8-49fa-b13d-c4720d55fa4a None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:07.384238 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7339486f-0fd8-49fa-b13d-c4720d55fa4a None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:07.384238 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7339486f-0fd8-49fa-b13d-c4720d55fa4a None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:07.385379 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 691/1395] 10.222.0.42 () {54 vars in 943 bytes} [Sun Sep 18 00:50:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:50:07.393138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c07a34-6101-400a-9040-4b648fdd6645 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:07.393595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c07a34-6101-400a-9040-4b648fdd6645 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:07.394014 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c07a34-6101-400a-9040-4b648fdd6645 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:07.474680 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:07.498783 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-64c07a34-6101-400a-9040-4b648fdd6645 None None] MFA Rules not processed for user `6230ed7ce0eb43e9b2b8530a28eb6299`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:07.500221 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-64c07a34-6101-400a-9040-4b648fdd6645 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:07.563359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ba780d-b9ec-48b1-84dd-7735951fc1a1 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:07.563665 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ba780d-b9ec-48b1-84dd-7735951fc1a1 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:07.563951 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ba780d-b9ec-48b1-84dd-7735951fc1a1 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:07.565466 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 692/1396] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:50:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:50:07.571331 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852ff66d-5755-4d8c-a98f-836991f3c20d None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:07.571625 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852ff66d-5755-4d8c-a98f-836991f3c20d None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:07.571872 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852ff66d-5755-4d8c-a98f-836991f3c20d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:07.588222 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:07.635681 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 705/1397] 10.222.0.42 () {58 vars in 1040 bytes} [Sun Sep 18 00:50:07 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:50:07.646307 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c07a34-6101-400a-9040-4b648fdd6645 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:50:07.653647 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-852ff66d-5755-4d8c-a98f-836991f3c20d None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:07.654971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-852ff66d-5755-4d8c-a98f-836991f3c20d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:07.656561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c07a34-6101-400a-9040-4b648fdd6645 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:50:07.775175 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:07.781636 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0caf6832-4e08-4082-8fc0-62a99520ca8c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6230ed7ce0eb43e9b2b8530a28eb6299', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:50:07.782170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 693/1398] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:50:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:50:07.784108 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0caf6832-4e08-4082-8fc0-62a99520ca8c service swift] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:07.784108 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0caf6832-4e08-4082-8fc0-62a99520ca8c service swift] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:07.784108 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0caf6832-4e08-4082-8fc0-62a99520ca8c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:07.784648 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-0caf6832-4e08-4082-8fc0-62a99520ca8c service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:07.891872 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0caf6832-4e08-4082-8fc0-62a99520ca8c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:50:07.893520 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0caf6832-4e08-4082-8fc0-62a99520ca8c service swift] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:50:07.902964 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 706/1399] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 18 00:50:07 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 257 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 18 00:50:08.106672 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852ff66d-5755-4d8c-a98f-836991f3c20d None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:50:08.108240 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-852ff66d-5755-4d8c-a98f-836991f3c20d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:08.257337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:08.304562 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852ff66d-5755-4d8c-a98f-836991f3c20d None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:50:08.505375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ad2ac1-479a-414b-91b7-bce493a81f5c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:50:08.506377 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ad2ac1-479a-414b-91b7-bce493a81f5c service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:08.506377 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ad2ac1-479a-414b-91b7-bce493a81f5c service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:08.506736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ad2ac1-479a-414b-91b7-bce493a81f5c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:08.508102 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d3ad2ac1-479a-414b-91b7-bce493a81f5c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:08.679728 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:08.685311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ad2ac1-479a-414b-91b7-bce493a81f5c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:50:08.687876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ad2ac1-479a-414b-91b7-bce493a81f5c service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:50:08.711043 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 694/1400] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:50:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 605 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:50:11.063938 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b1f6a1-b0c3-4e6d-a725-0a15a6718799 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:11.064560 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b1f6a1-b0c3-4e6d-a725-0a15a6718799 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:11.064670 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b1f6a1-b0c3-4e6d-a725-0a15a6718799 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:11.066435 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 707/1401] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:50:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:50:11.072910 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6f8d76-fb36-47f2-bbae-e4cdca88ce0c None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:11.073475 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6f8d76-fb36-47f2-bbae-e4cdca88ce0c None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:11.073475 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6f8d76-fb36-47f2-bbae-e4cdca88ce0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:11.171892 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-ce6f8d76-fb36-47f2-bbae-e4cdca88ce0c None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:11.173468 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ce6f8d76-fb36-47f2-bbae-e4cdca88ce0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:11.345523 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 695/1402] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:50:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 273 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:50:11.808234 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b1f6a1-b0c3-4e6d-a725-0a15a6718799 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:50:11.813772 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-44b1f6a1-b0c3-4e6d-a725-0a15a6718799 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:12.002421 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:12.017296 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b1f6a1-b0c3-4e6d-a725-0a15a6718799 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:50:12.206168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6328fd-a8bd-40d2-9f0a-409685909597 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:50:12.207236 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6328fd-a8bd-40d2-9f0a-409685909597 service nova] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:12.207236 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6328fd-a8bd-40d2-9f0a-409685909597 service nova] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:12.207527 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6328fd-a8bd-40d2-9f0a-409685909597 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:12.210019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-dc6328fd-a8bd-40d2-9f0a-409685909597 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:12.346769 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6328fd-a8bd-40d2-9f0a-409685909597 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:50:12.348587 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6328fd-a8bd-40d2-9f0a-409685909597 service nova] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:50:12.366214 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:12.380735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 708/1403] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:50:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 574 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:50:15.124559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd32d7a-91ce-4d93-a4d5-7243136cead6 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:15.125123 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd32d7a-91ce-4d93-a4d5-7243136cead6 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:15.125123 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd32d7a-91ce-4d93-a4d5-7243136cead6 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:15.127102 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 696/1404] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:50:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:50:15.133189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344fdc65-7ff0-4ac7-838b-3c4ff8691f0d None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:15.133713 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344fdc65-7ff0-4ac7-838b-3c4ff8691f0d None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:15.134113 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344fdc65-7ff0-4ac7-838b-3c4ff8691f0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:15.217068 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-344fdc65-7ff0-4ac7-838b-3c4ff8691f0d None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:15.218751 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-344fdc65-7ff0-4ac7-838b-3c4ff8691f0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:15.367460 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 709/1405] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:50:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:50:15.709936 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd32d7a-91ce-4d93-a4d5-7243136cead6 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:50:15.711978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-7fd32d7a-91ce-4d93-a4d5-7243136cead6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:15.834970 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:15.961258 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd32d7a-91ce-4d93-a4d5-7243136cead6 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:50:16.161137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32e75c8-27f7-4088-b9d1-01e2c6ba841c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:50:16.161811 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32e75c8-27f7-4088-b9d1-01e2c6ba841c service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:16.162063 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32e75c8-27f7-4088-b9d1-01e2c6ba841c service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:16.162313 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32e75c8-27f7-4088-b9d1-01e2c6ba841c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:16.164193 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-a32e75c8-27f7-4088-b9d1-01e2c6ba841c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:16.195996 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:16.288679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32e75c8-27f7-4088-b9d1-01e2c6ba841c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:50:16.290331 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32e75c8-27f7-4088-b9d1-01e2c6ba841c service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:50:16.318837 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 697/1406] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:50:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 610 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:50:18.682007 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c2b9ee-461f-4af2-bc07-c18c59173578 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:18.682604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c2b9ee-461f-4af2-bc07-c18c59173578 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:18.682604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c2b9ee-461f-4af2-bc07-c18c59173578 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:18.684110 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 710/1407] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:50:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:50:18.690012 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca4d9ab-091b-4f7e-97d3-299b89a1a305 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:18.690561 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca4d9ab-091b-4f7e-97d3-299b89a1a305 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:18.690788 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca4d9ab-091b-4f7e-97d3-299b89a1a305 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:18.768703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-9ca4d9ab-091b-4f7e-97d3-299b89a1a305 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:18.771287 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-9ca4d9ab-091b-4f7e-97d3-299b89a1a305 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:18.899407 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 698/1408] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:50:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:50:19.192570 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c2b9ee-461f-4af2-bc07-c18c59173578 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:50:19.194917 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-d3c2b9ee-461f-4af2-bc07-c18c59173578 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:19.285555 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:19.363984 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c2b9ee-461f-4af2-bc07-c18c59173578 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:50:19.499014 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0955970-b4be-426d-b5bf-fe07cd67469b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:50:19.499731 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0955970-b4be-426d-b5bf-fe07cd67469b service nova] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:19.500202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0955970-b4be-426d-b5bf-fe07cd67469b service nova] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:19.500476 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0955970-b4be-426d-b5bf-fe07cd67469b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:19.502144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c0955970-b4be-426d-b5bf-fe07cd67469b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:19.549143 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:19.620726 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0955970-b4be-426d-b5bf-fe07cd67469b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:50:19.622527 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0955970-b4be-426d-b5bf-fe07cd67469b service nova] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:50:19.647446 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 711/1409] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:50:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 453 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:50:28.895903 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:29.639534 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=29 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:31.415944 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381768af-f69e-4cca-ab05-3485add5205e None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:31.416499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381768af-f69e-4cca-ab05-3485add5205e None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:31.416499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381768af-f69e-4cca-ab05-3485add5205e None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:31.417954 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 699/1410] 10.222.0.42 () {54 vars in 918 bytes} [Sun Sep 18 00:50:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:50:31.423144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da6cd42-21ae-479f-8d8b-49e2e1fc1cb5 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:31.423620 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da6cd42-21ae-479f-8d8b-49e2e1fc1cb5 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:31.423724 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da6cd42-21ae-479f-8d8b-49e2e1fc1cb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:31.499268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-3da6cd42-21ae-479f-8d8b-49e2e1fc1cb5 None None] MFA Rules not processed for user `8f30116fb5ce43dab69efa8bafd9d049`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:31.500493 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3da6cd42-21ae-479f-8d8b-49e2e1fc1cb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:31.609333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 712/1411] 10.222.0.42 () {58 vars in 1015 bytes} [Sun Sep 18 00:50:31 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:50:31.624565 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d090e06-42ed-4d8f-af65-b50559f7ccf9 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:31.624856 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d090e06-42ed-4d8f-af65-b50559f7ccf9 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:31.625132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d090e06-42ed-4d8f-af65-b50559f7ccf9 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:31.626493 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 700/1412] 10.222.0.42 () {54 vars in 918 bytes} [Sun Sep 18 00:50:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:50:31.631799 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28636967-67bf-4600-9733-7555f8df4d33 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:31.632082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28636967-67bf-4600-9733-7555f8df4d33 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:31.632355 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28636967-67bf-4600-9733-7555f8df4d33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:31.708627 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:31.712147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-28636967-67bf-4600-9733-7555f8df4d33 None None] MFA Rules not processed for user `a59d73fd45a54651930d52300704fdb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:31.714061 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-28636967-67bf-4600-9733-7555f8df4d33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:31.830606 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 713/1413] 10.222.0.42 () {58 vars in 1015 bytes} [Sun Sep 18 00:50:31 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:50:34.489056 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e8e800-3e16-4022-aa12-4129bbb59864 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:34.489628 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e8e800-3e16-4022-aa12-4129bbb59864 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:34.489628 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e8e800-3e16-4022-aa12-4129bbb59864 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:34.491721 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 701/1414] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 18 00:50:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:50:34.498215 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d9394f-a858-4ebf-a6e5-1a9d58946f9a None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:34.498753 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d9394f-a858-4ebf-a6e5-1a9d58946f9a None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:34.498860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d9394f-a858-4ebf-a6e5-1a9d58946f9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:34.569306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-33d9394f-a858-4ebf-a6e5-1a9d58946f9a None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:34.570598 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-33d9394f-a858-4ebf-a6e5-1a9d58946f9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:34.626522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:34.672527 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 714/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 18 00:50:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:50:34.961797 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e8e800-3e16-4022-aa12-4129bbb59864 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:50:34.963438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-40e8e800-3e16-4022-aa12-4129bbb59864 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:35.109831 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e8e800-3e16-4022-aa12-4129bbb59864 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:50:35.186743 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:35.227389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e399b6-3bab-4ea4-9c6a-2d7109a6868b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:50:35.227897 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e399b6-3bab-4ea4-9c6a-2d7109a6868b service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:35.228176 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e399b6-3bab-4ea4-9c6a-2d7109a6868b service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:35.228451 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e399b6-3bab-4ea4-9c6a-2d7109a6868b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:35.230078 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-37e399b6-3bab-4ea4-9c6a-2d7109a6868b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:35.320495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e399b6-3bab-4ea4-9c6a-2d7109a6868b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:50:35.322164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e399b6-3bab-4ea4-9c6a-2d7109a6868b service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:50:35.340562 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 702/1416] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:50:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:50:36.524102 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3487bc3c-b1ad-4775-835e-27919ab35401 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:36.524807 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3487bc3c-b1ad-4775-835e-27919ab35401 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:36.524807 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3487bc3c-b1ad-4775-835e-27919ab35401 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:36.525976 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 715/1417] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 18 00:50:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:50:36.533248 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e925bb-2355-4595-9809-4f7891c4fffa None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:36.533835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e925bb-2355-4595-9809-4f7891c4fffa None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:36.533835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e925bb-2355-4595-9809-4f7891c4fffa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:36.618317 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-d7e925bb-2355-4595-9809-4f7891c4fffa None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:36.619576 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d7e925bb-2355-4595-9809-4f7891c4fffa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:36.647201 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:36.718034 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 703/1418] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 18 00:50:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:50:36.867078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3487bc3c-b1ad-4775-835e-27919ab35401 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:50:36.868584 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3487bc3c-b1ad-4775-835e-27919ab35401 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:37.019121 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3487bc3c-b1ad-4775-835e-27919ab35401 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:50:37.060394 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:37.135685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c79d73-cbfa-48f4-a0d8-394db959dbf5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:50:37.136213 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c79d73-cbfa-48f4-a0d8-394db959dbf5 service nova] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:37.136492 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c79d73-cbfa-48f4-a0d8-394db959dbf5 service nova] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:37.136762 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c79d73-cbfa-48f4-a0d8-394db959dbf5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:37.138400 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-78c79d73-cbfa-48f4-a0d8-394db959dbf5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:37.229166 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78c79d73-cbfa-48f4-a0d8-394db959dbf5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:50:37.230801 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78c79d73-cbfa-48f4-a0d8-394db959dbf5 service nova] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:50:37.250180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 716/1419] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:50:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:50:38.412625 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da9e9c6-8b71-46e9-9b3a-deb53d93a979 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:38.413164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da9e9c6-8b71-46e9-9b3a-deb53d93a979 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:38.413164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da9e9c6-8b71-46e9-9b3a-deb53d93a979 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:38.414649 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 704/1420] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 18 00:50:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:50:38.421868 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50920ddb-fda3-442c-bd61-8bdc499c130f None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:38.422351 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50920ddb-fda3-442c-bd61-8bdc499c130f None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:38.422488 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50920ddb-fda3-442c-bd61-8bdc499c130f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:38.478153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:38.503272 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-50920ddb-fda3-442c-bd61-8bdc499c130f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:38.504446 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-50920ddb-fda3-442c-bd61-8bdc499c130f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:38.602026 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 717/1421] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 18 00:50:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:50:38.754056 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da9e9c6-8b71-46e9-9b3a-deb53d93a979 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:50:38.755617 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-0da9e9c6-8b71-46e9-9b3a-deb53d93a979 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:38.925689 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da9e9c6-8b71-46e9-9b3a-deb53d93a979 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:50:38.938492 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:39.040371 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c574511-e901-4776-be8c-e49a464332af service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:50:39.040977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c574511-e901-4776-be8c-e49a464332af service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:39.041258 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c574511-e901-4776-be8c-e49a464332af service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:39.041532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c574511-e901-4776-be8c-e49a464332af service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:39.043181 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-5c574511-e901-4776-be8c-e49a464332af service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:39.141184 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c574511-e901-4776-be8c-e49a464332af service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:50:39.142824 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c574511-e901-4776-be8c-e49a464332af service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:50:39.167717 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 705/1422] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:50:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:50:40.307690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74d295c-85c9-4ed4-81cb-768062333004 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:40.308246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74d295c-85c9-4ed4-81cb-768062333004 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:40.308246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74d295c-85c9-4ed4-81cb-768062333004 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:40.309599 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 718/1423] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 18 00:50:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:50:40.316326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf932b8-4fd3-4aec-bf93-dd3b1f7bd47e None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:40.316820 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf932b8-4fd3-4aec-bf93-dd3b1f7bd47e None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:40.316922 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf932b8-4fd3-4aec-bf93-dd3b1f7bd47e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:40.336416 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:40.393694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-1bf932b8-4fd3-4aec-bf93-dd3b1f7bd47e None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:40.394852 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-1bf932b8-4fd3-4aec-bf93-dd3b1f7bd47e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:40.490826 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 706/1424] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 18 00:50:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:50:40.644634 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74d295c-85c9-4ed4-81cb-768062333004 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:50:40.646106 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-b74d295c-85c9-4ed4-81cb-768062333004 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:40.756652 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:40.792523 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74d295c-85c9-4ed4-81cb-768062333004 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:50:40.906048 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa94f9c-d289-4762-8be5-d4536921a5b5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:50:40.906598 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa94f9c-d289-4762-8be5-d4536921a5b5 service nova] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:40.906875 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa94f9c-d289-4762-8be5-d4536921a5b5 service nova] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:40.907146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa94f9c-d289-4762-8be5-d4536921a5b5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:40.908766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-caa94f9c-d289-4762-8be5-d4536921a5b5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:40.987491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:41.002223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa94f9c-d289-4762-8be5-d4536921a5b5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:50:41.003913 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa94f9c-d289-4762-8be5-d4536921a5b5 service nova] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:50:41.022211 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 719/1425] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:50:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:50:42.165296 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae5d435-703e-4da2-a654-ada52c702a03 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:42.165909 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae5d435-703e-4da2-a654-ada52c702a03 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:42.165909 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae5d435-703e-4da2-a654-ada52c702a03 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:42.167220 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 707/1426] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 18 00:50:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:50:42.173638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955d3655-bd79-4491-aeaf-d7bdc9320f52 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:42.174095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955d3655-bd79-4491-aeaf-d7bdc9320f52 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:42.174194 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955d3655-bd79-4491-aeaf-d7bdc9320f52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:42.248942 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-955d3655-bd79-4491-aeaf-d7bdc9320f52 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:42.250131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-955d3655-bd79-4491-aeaf-d7bdc9320f52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:42.347120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 720/1427] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 18 00:50:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:50:42.499496 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae5d435-703e-4da2-a654-ada52c702a03 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:50:42.500961 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-cae5d435-703e-4da2-a654-ada52c702a03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:42.585779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:42.648768 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae5d435-703e-4da2-a654-ada52c702a03 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:50:42.761833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05aeb3be-06ad-438f-bdd8-75b800752320 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:50:42.762374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05aeb3be-06ad-438f-bdd8-75b800752320 service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:42.762674 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05aeb3be-06ad-438f-bdd8-75b800752320 service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:42.762939 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05aeb3be-06ad-438f-bdd8-75b800752320 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:42.764515 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-05aeb3be-06ad-438f-bdd8-75b800752320 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:42.813630 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:42.855904 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05aeb3be-06ad-438f-bdd8-75b800752320 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:50:42.857504 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05aeb3be-06ad-438f-bdd8-75b800752320 service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:50:42.875202 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 708/1428] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:50:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:50:43.951738 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090820fe-24a7-4aa5-b67c-d827320810da None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:43.952289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090820fe-24a7-4aa5-b67c-d827320810da None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:43.952289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090820fe-24a7-4aa5-b67c-d827320810da None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:43.953792 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 721/1429] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 18 00:50:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:50:43.962363 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd9ea88-1dd0-4f20-9310-1ea28844c7f0 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:43.962908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd9ea88-1dd0-4f20-9310-1ea28844c7f0 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:43.963000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd9ea88-1dd0-4f20-9310-1ea28844c7f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:44.038971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-5fd9ea88-1dd0-4f20-9310-1ea28844c7f0 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:44.040198 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-5fd9ea88-1dd0-4f20-9310-1ea28844c7f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:44.134194 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 709/1430] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 18 00:50:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:50:44.288330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090820fe-24a7-4aa5-b67c-d827320810da None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:50:44.289889 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-090820fe-24a7-4aa5-b67c-d827320810da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:44.339194 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:44.438642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090820fe-24a7-4aa5-b67c-d827320810da None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:50:44.561282 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c582199e-196e-41ae-a61a-87c6cceabc75 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:50:44.561956 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c582199e-196e-41ae-a61a-87c6cceabc75 service nova] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:44.562180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c582199e-196e-41ae-a61a-87c6cceabc75 service nova] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:44.562482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c582199e-196e-41ae-a61a-87c6cceabc75 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:44.564130 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c582199e-196e-41ae-a61a-87c6cceabc75 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:44.571730 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:44.663110 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c582199e-196e-41ae-a61a-87c6cceabc75 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:50:44.664745 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c582199e-196e-41ae-a61a-87c6cceabc75 service nova] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:50:44.683782 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 722/1431] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:50:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:50:45.754578 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d0534e-4d0c-41d9-b532-881bc4816836 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:45.755138 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d0534e-4d0c-41d9-b532-881bc4816836 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:45.755138 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d0534e-4d0c-41d9-b532-881bc4816836 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:45.756475 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 710/1432] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 18 00:50:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:50:45.762919 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8ff7d7-7486-4b41-a981-1beee386ad25 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:45.763413 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8ff7d7-7486-4b41-a981-1beee386ad25 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:45.763517 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8ff7d7-7486-4b41-a981-1beee386ad25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:45.843184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-aa8ff7d7-7486-4b41-a981-1beee386ad25 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:45.844406 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-aa8ff7d7-7486-4b41-a981-1beee386ad25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:45.916236 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:45.947541 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 723/1433] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 18 00:50:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:50:46.098959 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d0534e-4d0c-41d9-b532-881bc4816836 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:50:46.100969 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-46d0534e-4d0c-41d9-b532-881bc4816836 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:46.118597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:46.248704 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d0534e-4d0c-41d9-b532-881bc4816836 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:50:46.337308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:46.365262 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be33ca4d-cf7c-4460-930b-a88f149bdd54 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:50:46.365778 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be33ca4d-cf7c-4460-930b-a88f149bdd54 service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:46.366052 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be33ca4d-cf7c-4460-930b-a88f149bdd54 service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:46.366321 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be33ca4d-cf7c-4460-930b-a88f149bdd54 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:46.367959 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-be33ca4d-cf7c-4460-930b-a88f149bdd54 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:46.459370 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be33ca4d-cf7c-4460-930b-a88f149bdd54 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:50:46.460971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be33ca4d-cf7c-4460-930b-a88f149bdd54 service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:50:46.478606 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 711/1434] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:50:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:50:47.536877 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78769d59-a510-4cf0-bbc3-7e4c392f6d45 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:47.537469 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78769d59-a510-4cf0-bbc3-7e4c392f6d45 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:47.537469 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78769d59-a510-4cf0-bbc3-7e4c392f6d45 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:47.538858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 724/1435] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 18 00:50:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:50:47.546145 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd2e75a-7581-452c-99bc-b05df8c3bcbd None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:47.546707 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd2e75a-7581-452c-99bc-b05df8c3bcbd None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:47.546707 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd2e75a-7581-452c-99bc-b05df8c3bcbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:47.623503 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-8fd2e75a-7581-452c-99bc-b05df8c3bcbd None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:47.624741 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8fd2e75a-7581-452c-99bc-b05df8c3bcbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:47.667881 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:47.726977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 712/1436] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 18 00:50:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:50:47.876385 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78769d59-a510-4cf0-bbc3-7e4c392f6d45 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:50:47.877882 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-78769d59-a510-4cf0-bbc3-7e4c392f6d45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:48.021416 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78769d59-a510-4cf0-bbc3-7e4c392f6d45 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:50:48.076473 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:48.134572 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67b22bc-dbba-4126-8343-cce0f5961f8e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:50:48.135189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67b22bc-dbba-4126-8343-cce0f5961f8e service nova] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:48.135564 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67b22bc-dbba-4126-8343-cce0f5961f8e service nova] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:48.135903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67b22bc-dbba-4126-8343-cce0f5961f8e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:48.137545 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a67b22bc-dbba-4126-8343-cce0f5961f8e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:48.232771 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67b22bc-dbba-4126-8343-cce0f5961f8e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:50:48.236298 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67b22bc-dbba-4126-8343-cce0f5961f8e service nova] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:50:48.255078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 725/1437] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:50:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:50:49.329121 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d092209e-40db-4904-9ac7-5549f8bd422b None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:49.329669 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d092209e-40db-4904-9ac7-5549f8bd422b None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:49.329669 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d092209e-40db-4904-9ac7-5549f8bd422b None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:49.331073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 713/1438] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 18 00:50:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:50:49.338359 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06259f32-4eb5-48a0-b1ae-b8101fe52db3 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:49.338952 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06259f32-4eb5-48a0-b1ae-b8101fe52db3 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:49.338952 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06259f32-4eb5-48a0-b1ae-b8101fe52db3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:49.422858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-06259f32-4eb5-48a0-b1ae-b8101fe52db3 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:49.424107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-06259f32-4eb5-48a0-b1ae-b8101fe52db3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:49.426682 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:49.524741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 726/1439] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 18 00:50:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:50:49.680271 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d092209e-40db-4904-9ac7-5549f8bd422b None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:50:49.681696 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d092209e-40db-4904-9ac7-5549f8bd422b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:49.826730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d092209e-40db-4904-9ac7-5549f8bd422b None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:50:49.853883 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:49.948038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c1173e-6a21-404f-93e8-eb029b02c0ca service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:50:49.948567 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c1173e-6a21-404f-93e8-eb029b02c0ca service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:49.948849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c1173e-6a21-404f-93e8-eb029b02c0ca service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:49.949121 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c1173e-6a21-404f-93e8-eb029b02c0ca service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:49.950755 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-37c1173e-6a21-404f-93e8-eb029b02c0ca service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:50.043084 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c1173e-6a21-404f-93e8-eb029b02c0ca service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:50:50.044680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c1173e-6a21-404f-93e8-eb029b02c0ca service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:50:50.062777 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 714/1440] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:50:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:50:52.137218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7e27ca-266d-412e-9f02-a9725b0ae2cc None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:52.137825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7e27ca-266d-412e-9f02-a9725b0ae2cc None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:52.137825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7e27ca-266d-412e-9f02-a9725b0ae2cc None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:52.139145 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 727/1441] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 18 00:50:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:50:52.145064 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f996ad-453e-41f5-93b3-02d0928d2b54 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:52.145572 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f996ad-453e-41f5-93b3-02d0928d2b54 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:52.145660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f996ad-453e-41f5-93b3-02d0928d2b54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:52.188938 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:52.218845 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-91f996ad-453e-41f5-93b3-02d0928d2b54 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:52.220167 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-91f996ad-453e-41f5-93b3-02d0928d2b54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:52.325539 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 715/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 18 00:50:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:50:52.607839 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7e27ca-266d-412e-9f02-a9725b0ae2cc None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:50:52.609377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-5f7e27ca-266d-412e-9f02-a9725b0ae2cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:52.714115 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7e27ca-266d-412e-9f02-a9725b0ae2cc None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:50:52.745997 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:52.788825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7c8f40-0ac6-4f93-a055-157359359674 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:50:52.789369 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7c8f40-0ac6-4f93-a055-157359359674 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:52.789638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7c8f40-0ac6-4f93-a055-157359359674 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:52.789902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7c8f40-0ac6-4f93-a055-157359359674 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:52.791531 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-cf7c8f40-0ac6-4f93-a055-157359359674 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:52.901718 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7c8f40-0ac6-4f93-a055-157359359674 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:50:52.903191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7c8f40-0ac6-4f93-a055-157359359674 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:50:52.922018 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 728/1443] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:50:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:50:55.285497 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705c5051-5b9f-40e7-95fe-52066cfa8ef8 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:55.286086 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705c5051-5b9f-40e7-95fe-52066cfa8ef8 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:55.286086 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705c5051-5b9f-40e7-95fe-52066cfa8ef8 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:55.289468 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 716/1444] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 18 00:50:55 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:50:55.296690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08063c62-2ca6-4026-a575-0a7dfeb9a151 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:55.297261 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08063c62-2ca6-4026-a575-0a7dfeb9a151 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:55.297261 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08063c62-2ca6-4026-a575-0a7dfeb9a151 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:55.372677 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-08063c62-2ca6-4026-a575-0a7dfeb9a151 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:55.373912 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-08063c62-2ca6-4026-a575-0a7dfeb9a151 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:55.387791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:55.473479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 729/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 18 00:50:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:50:55.757411 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705c5051-5b9f-40e7-95fe-52066cfa8ef8 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:50:55.758978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-705c5051-5b9f-40e7-95fe-52066cfa8ef8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:55.862248 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705c5051-5b9f-40e7-95fe-52066cfa8ef8 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:50:55.875305 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:55.937278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee6f411-08e6-4e0c-a487-f9be0b1b0deb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:50:55.937882 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee6f411-08e6-4e0c-a487-f9be0b1b0deb service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:55.938161 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee6f411-08e6-4e0c-a487-f9be0b1b0deb service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:55.938423 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee6f411-08e6-4e0c-a487-f9be0b1b0deb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:55.940043 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-2ee6f411-08e6-4e0c-a487-f9be0b1b0deb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:56.050525 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee6f411-08e6-4e0c-a487-f9be0b1b0deb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:50:56.052019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee6f411-08e6-4e0c-a487-f9be0b1b0deb service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:50:56.189554 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 717/1446] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:50:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:50:58.793172 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758eaa89-cdb2-465a-989f-5b2c7a75d0da None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:58.793679 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758eaa89-cdb2-465a-989f-5b2c7a75d0da None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:58.793679 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758eaa89-cdb2-465a-989f-5b2c7a75d0da None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:58.795011 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 730/1447] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 18 00:50:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:50:58.800405 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b48559-3aee-4bb5-aede-0bb1453df5f4 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:58.800844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b48559-3aee-4bb5-aede-0bb1453df5f4 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:58.800940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b48559-3aee-4bb5-aede-0bb1453df5f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:58.848233 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:58.867878 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-56b48559-3aee-4bb5-aede-0bb1453df5f4 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:50:58.869109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-56b48559-3aee-4bb5-aede-0bb1453df5f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:58.967218 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 718/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 18 00:50:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:50:59.252694 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758eaa89-cdb2-465a-989f-5b2c7a75d0da None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:50:59.254235 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-758eaa89-cdb2-465a-989f-5b2c7a75d0da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:59.357954 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758eaa89-cdb2-465a-989f-5b2c7a75d0da None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:50:59.403589 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:50:59.440011 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acaccd7-14a8-46e1-a48d-7d0fde6e1809 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:50:59.440655 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acaccd7-14a8-46e1-a48d-7d0fde6e1809 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:50:59.440938 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acaccd7-14a8-46e1-a48d-7d0fde6e1809 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:50:59.441211 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acaccd7-14a8-46e1-a48d-7d0fde6e1809 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:50:59.442852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-8acaccd7-14a8-46e1-a48d-7d0fde6e1809 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:50:59.555783 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8acaccd7-14a8-46e1-a48d-7d0fde6e1809 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:50:59.557218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8acaccd7-14a8-46e1-a48d-7d0fde6e1809 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:50:59.575537 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 731/1449] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:50:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:51:01.873227 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50db89ac-d2f9-4519-9750-5e5c22a83eac None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:01.874016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50db89ac-d2f9-4519-9750-5e5c22a83eac None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:01.874016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50db89ac-d2f9-4519-9750-5e5c22a83eac None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:01.875162 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 719/1450] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 18 00:51:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:51:01.880612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf09156-8c8a-4ef3-a075-a1f4dc55cb7e None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:01.881016 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf09156-8c8a-4ef3-a075-a1f4dc55cb7e None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:01.881248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf09156-8c8a-4ef3-a075-a1f4dc55cb7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:01.948024 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-bdf09156-8c8a-4ef3-a075-a1f4dc55cb7e None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:51:01.949257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-bdf09156-8c8a-4ef3-a075-a1f4dc55cb7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:01.971163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:02.048376 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 732/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 18 00:51:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:51:02.330271 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50db89ac-d2f9-4519-9750-5e5c22a83eac None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:51:02.331867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-50db89ac-d2f9-4519-9750-5e5c22a83eac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:02.444138 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50db89ac-d2f9-4519-9750-5e5c22a83eac None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:51:02.465476 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:02.522213 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa506e1f-d4e9-4fd0-a175-b1e1a9a57ca0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:51:02.522849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa506e1f-d4e9-4fd0-a175-b1e1a9a57ca0 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:02.523130 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa506e1f-d4e9-4fd0-a175-b1e1a9a57ca0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:02.523404 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa506e1f-d4e9-4fd0-a175-b1e1a9a57ca0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:02.525040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-aa506e1f-d4e9-4fd0-a175-b1e1a9a57ca0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:02.634195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa506e1f-d4e9-4fd0-a175-b1e1a9a57ca0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:51:02.635671 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa506e1f-d4e9-4fd0-a175-b1e1a9a57ca0 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:51:02.653932 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 720/1452] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:51:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:51:04.928067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd593e5-be16-4b74-9cab-440b4076a2f2 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:04.928603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd593e5-be16-4b74-9cab-440b4076a2f2 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:04.928603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd593e5-be16-4b74-9cab-440b4076a2f2 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:04.929954 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 733/1453] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 18 00:51:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:51:04.935492 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0af475-2308-416d-85ca-6d3d4c4553ad None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:04.935970 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0af475-2308-416d-85ca-6d3d4c4553ad None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:04.936319 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0af475-2308-416d-85ca-6d3d4c4553ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:05.021836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:05.025082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-8c0af475-2308-416d-85ca-6d3d4c4553ad None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:51:05.027005 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8c0af475-2308-416d-85ca-6d3d4c4553ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:05.125846 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 721/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 18 00:51:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:51:05.420166 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd593e5-be16-4b74-9cab-440b4076a2f2 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:51:05.421722 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-afd593e5-be16-4b74-9cab-440b4076a2f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:05.527998 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd593e5-be16-4b74-9cab-440b4076a2f2 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:51:05.579247 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:05.605374 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce892d7-dce4-41cd-8f7c-9c1cfa4b0a5e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:51:05.605975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce892d7-dce4-41cd-8f7c-9c1cfa4b0a5e service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:05.606250 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce892d7-dce4-41cd-8f7c-9c1cfa4b0a5e service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:05.606545 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce892d7-dce4-41cd-8f7c-9c1cfa4b0a5e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:05.608124 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-2ce892d7-dce4-41cd-8f7c-9c1cfa4b0a5e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:05.716768 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce892d7-dce4-41cd-8f7c-9c1cfa4b0a5e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:51:05.718113 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce892d7-dce4-41cd-8f7c-9c1cfa4b0a5e service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:51:05.735682 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 734/1455] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:51:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:51:07.981898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76a85f1-ec94-424c-bc22-ae1db2a26cbd None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:07.982446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76a85f1-ec94-424c-bc22-ae1db2a26cbd None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:07.982446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76a85f1-ec94-424c-bc22-ae1db2a26cbd None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:07.983811 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 722/1456] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 18 00:51:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:51:07.989179 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48996d4f-a04b-4778-bdc5-37b023dacb9c None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:07.989635 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48996d4f-a04b-4778-bdc5-37b023dacb9c None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:07.989873 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48996d4f-a04b-4778-bdc5-37b023dacb9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:08.056559 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-48996d4f-a04b-4778-bdc5-37b023dacb9c None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:51:08.057781 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-48996d4f-a04b-4778-bdc5-37b023dacb9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:08.087003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:08.153630 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 735/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 18 00:51:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:51:08.441709 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76a85f1-ec94-424c-bc22-ae1db2a26cbd None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:51:08.443272 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-b76a85f1-ec94-424c-bc22-ae1db2a26cbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:08.553319 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76a85f1-ec94-424c-bc22-ae1db2a26cbd None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:51:08.581628 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:08.628005 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54cc9291-7c96-4a91-b111-14d2d577cbf8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:51:08.628624 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cc9291-7c96-4a91-b111-14d2d577cbf8 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:08.628901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cc9291-7c96-4a91-b111-14d2d577cbf8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:08.629173 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cc9291-7c96-4a91-b111-14d2d577cbf8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:08.631416 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-54cc9291-7c96-4a91-b111-14d2d577cbf8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:08.741436 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54cc9291-7c96-4a91-b111-14d2d577cbf8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:51:08.742957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54cc9291-7c96-4a91-b111-14d2d577cbf8 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:51:08.760810 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 723/1458] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:51:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:51:10.976614 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2990f1-33d0-4358-976b-b5d9f8ad7f9b None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:10.977211 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2990f1-33d0-4358-976b-b5d9f8ad7f9b None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:10.977315 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2990f1-33d0-4358-976b-b5d9f8ad7f9b None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:10.978925 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 736/1459] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 18 00:51:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:51:10.984672 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b68a86-fb88-4e7b-9a3f-b9f777f685b2 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:10.985196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b68a86-fb88-4e7b-9a3f-b9f777f685b2 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:10.985196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b68a86-fb88-4e7b-9a3f-b9f777f685b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:11.051642 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-75b68a86-fb88-4e7b-9a3f-b9f777f685b2 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:51:11.053482 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-75b68a86-fb88-4e7b-9a3f-b9f777f685b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:11.061968 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:11.149739 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 724/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 18 00:51:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:51:11.438538 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2990f1-33d0-4358-976b-b5d9f8ad7f9b None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:51:11.440070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-5c2990f1-33d0-4358-976b-b5d9f8ad7f9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:11.542894 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2990f1-33d0-4358-976b-b5d9f8ad7f9b None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:51:11.600910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:11.619212 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54119c30-e433-433d-ba61-7dbc22f354a4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:51:11.619823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54119c30-e433-433d-ba61-7dbc22f354a4 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:11.620102 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54119c30-e433-433d-ba61-7dbc22f354a4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:11.620374 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54119c30-e433-433d-ba61-7dbc22f354a4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:11.621981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-54119c30-e433-433d-ba61-7dbc22f354a4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:11.730397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54119c30-e433-433d-ba61-7dbc22f354a4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:51:11.731792 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54119c30-e433-433d-ba61-7dbc22f354a4 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:51:11.750064 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 737/1461] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:51:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:51:13.954292 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4b31f4-f68b-43a7-b4c1-8b1d224624fa None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:13.954941 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4b31f4-f68b-43a7-b4c1-8b1d224624fa None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:13.954941 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4b31f4-f68b-43a7-b4c1-8b1d224624fa None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:13.956231 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 725/1462] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 18 00:51:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:51:13.962207 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba65585-f566-4bb8-b4f2-66aeb72fa58c None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:13.962740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba65585-f566-4bb8-b4f2-66aeb72fa58c None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:13.962844 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba65585-f566-4bb8-b4f2-66aeb72fa58c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:14.028245 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-3ba65585-f566-4bb8-b4f2-66aeb72fa58c None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:51:14.029449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3ba65585-f566-4bb8-b4f2-66aeb72fa58c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:14.067726 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:14.126233 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 738/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 18 00:51:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:51:14.411367 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4b31f4-f68b-43a7-b4c1-8b1d224624fa None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:51:14.413841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-4c4b31f4-f68b-43a7-b4c1-8b1d224624fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:14.522423 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4b31f4-f68b-43a7-b4c1-8b1d224624fa None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:51:14.561558 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:14.598371 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcfc31f-f3e4-4a7d-92a3-8ee401ae87c0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:51:14.599568 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcfc31f-f3e4-4a7d-92a3-8ee401ae87c0 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:14.599848 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcfc31f-f3e4-4a7d-92a3-8ee401ae87c0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:14.600123 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcfc31f-f3e4-4a7d-92a3-8ee401ae87c0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:14.601736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-cbcfc31f-f3e4-4a7d-92a3-8ee401ae87c0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:14.711108 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcfc31f-f3e4-4a7d-92a3-8ee401ae87c0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:51:14.712550 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcfc31f-f3e4-4a7d-92a3-8ee401ae87c0 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:51:14.730727 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 726/1464] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:51:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:51:16.914906 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b9d1b4-0458-44e7-995e-1ea5aeb306ee None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:16.915478 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b9d1b4-0458-44e7-995e-1ea5aeb306ee None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:16.915478 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b9d1b4-0458-44e7-995e-1ea5aeb306ee None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:16.916797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 739/1465] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 18 00:51:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:51:16.922322 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0be091-c403-4adf-827f-2401682b13e4 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:16.922828 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0be091-c403-4adf-827f-2401682b13e4 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:16.922937 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0be091-c403-4adf-827f-2401682b13e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:16.989890 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-ca0be091-c403-4adf-827f-2401682b13e4 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:51:16.991068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ca0be091-c403-4adf-827f-2401682b13e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:17.010491 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:17.091115 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 727/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 18 00:51:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:51:17.371374 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b9d1b4-0458-44e7-995e-1ea5aeb306ee None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:51:17.372905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a5b9d1b4-0458-44e7-995e-1ea5aeb306ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:17.481069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b9d1b4-0458-44e7-995e-1ea5aeb306ee None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:51:17.548929 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:17.556591 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3563c4f4-b995-4961-a01b-6f9305b0ae90 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:51:17.557200 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3563c4f4-b995-4961-a01b-6f9305b0ae90 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:17.557476 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3563c4f4-b995-4961-a01b-6f9305b0ae90 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:17.557748 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3563c4f4-b995-4961-a01b-6f9305b0ae90 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:17.559367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3563c4f4-b995-4961-a01b-6f9305b0ae90 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:17.677561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3563c4f4-b995-4961-a01b-6f9305b0ae90 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:51:17.678962 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3563c4f4-b995-4961-a01b-6f9305b0ae90 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:51:17.814612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 740/1467] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:51:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 444 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:51:19.991829 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f9ea7e-7711-443f-969f-ccac57748f83 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:19.992422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f9ea7e-7711-443f-969f-ccac57748f83 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:19.992547 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f9ea7e-7711-443f-969f-ccac57748f83 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:19.993948 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 728/1468] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 18 00:51:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:51:19.999957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662e61b9-5104-4656-8607-c11fc5b84cee None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:20.000451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662e61b9-5104-4656-8607-c11fc5b84cee None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:20.000553 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662e61b9-5104-4656-8607-c11fc5b84cee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:20.077728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-662e61b9-5104-4656-8607-c11fc5b84cee None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:51:20.078979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-662e61b9-5104-4656-8607-c11fc5b84cee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:20.131797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:20.181589 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 741/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 18 00:51:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:51:20.470149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f9ea7e-7711-443f-969f-ccac57748f83 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:51:20.471737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-43f9ea7e-7711-443f-969f-ccac57748f83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:20.584404 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f9ea7e-7711-443f-969f-ccac57748f83 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:51:20.630927 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:20.656382 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde2bf13-db2d-42ea-83ef-dabae535a9ad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:51:20.656979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde2bf13-db2d-42ea-83ef-dabae535a9ad service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:20.657253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde2bf13-db2d-42ea-83ef-dabae535a9ad service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:20.657521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde2bf13-db2d-42ea-83ef-dabae535a9ad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:20.659131 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-fde2bf13-db2d-42ea-83ef-dabae535a9ad service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:20.767509 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde2bf13-db2d-42ea-83ef-dabae535a9ad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:51:20.768940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde2bf13-db2d-42ea-83ef-dabae535a9ad service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:51:20.787921 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 729/1470] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:51:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:51:30.180014 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=22 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:30.786198 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=66 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:41.028099 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc122af6-1d5b-41d7-baaf-30eeb7a06488 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:41.028670 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc122af6-1d5b-41d7-baaf-30eeb7a06488 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:41.028670 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc122af6-1d5b-41d7-baaf-30eeb7a06488 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:41.030072 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 742/1471] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:51:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:51:41.036025 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dad85c-3919-4ffa-94ac-55877c5c05b7 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:41.036454 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dad85c-3919-4ffa-94ac-55877c5c05b7 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:41.036593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dad85c-3919-4ffa-94ac-55877c5c05b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:41.108529 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-81dad85c-3919-4ffa-94ac-55877c5c05b7 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:51:41.109869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-81dad85c-3919-4ffa-94ac-55877c5c05b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:41.209956 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 730/1472] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:51:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:51:41.507449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7276e008-86cf-41d4-b7d9-d1bc98a4e4f3 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:41.507717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7276e008-86cf-41d4-b7d9-d1bc98a4e4f3 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:41.507991 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7276e008-86cf-41d4-b7d9-d1bc98a4e4f3 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:41.509325 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 743/1473] 10.222.0.42 () {54 vars in 962 bytes} [Sun Sep 18 00:51:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:51:41.514655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8528ef63-1c20-4c11-8504-8fb2b43cd907 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:41.514944 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8528ef63-1c20-4c11-8504-8fb2b43cd907 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:41.515220 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8528ef63-1c20-4c11-8504-8fb2b43cd907 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:41.591862 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-8528ef63-1c20-4c11-8504-8fb2b43cd907 None None] MFA Rules not processed for user `d7fe9d21099e4c4891c28687bb673214`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:51:41.593074 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8528ef63-1c20-4c11-8504-8fb2b43cd907 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:41.614278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:41.668484 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 731/1474] 10.222.0.42 () {58 vars in 1059 bytes} [Sun Sep 18 00:51:41 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:51:41.677327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7276e008-86cf-41d4-b7d9-d1bc98a4e4f3 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:51:41.678900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-7276e008-86cf-41d4-b7d9-d1bc98a4e4f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:41.783970 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7276e008-86cf-41d4-b7d9-d1bc98a4e4f3 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:51:41.856159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b4d67d-76f3-4326-ba17-e049baf42367 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7fe9d21099e4c4891c28687bb673214', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:51:41.856846 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b4d67d-76f3-4326-ba17-e049baf42367 service glance] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:41.857043 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b4d67d-76f3-4326-ba17-e049baf42367 service glance] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:41.857307 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b4d67d-76f3-4326-ba17-e049baf42367 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:41.858914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-84b4d67d-76f3-4326-ba17-e049baf42367 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:41.913197 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:41.968291 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b4d67d-76f3-4326-ba17-e049baf42367 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:51:41.969733 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b4d67d-76f3-4326-ba17-e049baf42367 service glance] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:51:41.987781 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 744/1475] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:51:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:51:44.274842 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe04e7d-48b7-444c-9bac-4cf3c7ca8950 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:44.275475 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe04e7d-48b7-444c-9bac-4cf3c7ca8950 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:44.275475 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe04e7d-48b7-444c-9bac-4cf3c7ca8950 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:44.276770 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 732/1476] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:51:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:51:44.282267 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412c2df2-c81f-4df9-a248-bb06c2460cef None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:44.282756 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412c2df2-c81f-4df9-a248-bb06c2460cef None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:44.282860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412c2df2-c81f-4df9-a248-bb06c2460cef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:44.349862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-412c2df2-c81f-4df9-a248-bb06c2460cef None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:51:44.351095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-412c2df2-c81f-4df9-a248-bb06c2460cef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:44.451772 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 745/1477] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:51:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:51:44.734387 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe04e7d-48b7-444c-9bac-4cf3c7ca8950 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:51:44.736035 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ffe04e7d-48b7-444c-9bac-4cf3c7ca8950 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:44.899697 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe04e7d-48b7-444c-9bac-4cf3c7ca8950 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:51:44.955681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:45.002030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3dbb661-db8c-4e51-97a2-e22a65c477b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:51:45.002582 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dbb661-db8c-4e51-97a2-e22a65c477b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:45.002862 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dbb661-db8c-4e51-97a2-e22a65c477b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:45.003133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dbb661-db8c-4e51-97a2-e22a65c477b6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:45.003773 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3dbb661-db8c-4e51-97a2-e22a65c477b6 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:51:45.005572 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: WARNING py.warnings [None req-d3dbb661-db8c-4e51-97a2-e22a65c477b6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 18 00:51:45.005572 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: warnings.warn(msg) Sep 18 00:51:45.005572 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: Sep 18 00:51:45.006731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3dbb661-db8c-4e51-97a2-e22a65c477b6 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:51:45.026446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 733/1478] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 18 00:51:44 2022] GET /identity/v3/projects => generated 2553 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:51:47.219050 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c733fe5-8c1c-4979-bbdc-f842bd99466a None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:47.219620 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c733fe5-8c1c-4979-bbdc-f842bd99466a None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:47.219620 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c733fe5-8c1c-4979-bbdc-f842bd99466a None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:47.220932 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 746/1479] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:51:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:51:47.227145 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc78963a-795b-4f9a-8344-f4e322223739 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:47.227571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc78963a-795b-4f9a-8344-f4e322223739 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:47.227692 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc78963a-795b-4f9a-8344-f4e322223739 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:47.293067 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-cc78963a-795b-4f9a-8344-f4e322223739 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:51:47.294194 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-cc78963a-795b-4f9a-8344-f4e322223739 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:47.391036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 734/1480] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:51:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:51:47.701913 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c733fe5-8c1c-4979-bbdc-f842bd99466a None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:51:47.703470 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-9c733fe5-8c1c-4979-bbdc-f842bd99466a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:47.714103 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:47.856969 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c733fe5-8c1c-4979-bbdc-f842bd99466a None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:51:47.935645 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:47.967670 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74cddc7d-ff19-4926-a688-30086d9fed23 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:51:47.968196 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74cddc7d-ff19-4926-a688-30086d9fed23 service nova] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:47.968473 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74cddc7d-ff19-4926-a688-30086d9fed23 service nova] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:47.968744 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74cddc7d-ff19-4926-a688-30086d9fed23 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:47.970320 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-74cddc7d-ff19-4926-a688-30086d9fed23 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:48.061081 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74cddc7d-ff19-4926-a688-30086d9fed23 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:51:48.062976 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74cddc7d-ff19-4926-a688-30086d9fed23 service nova] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:51:48.080764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 747/1481] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:51:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:51:50.425155 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1732358-d7cb-4e09-9a3b-06a19b67a519 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:50.425707 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1732358-d7cb-4e09-9a3b-06a19b67a519 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:50.425707 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1732358-d7cb-4e09-9a3b-06a19b67a519 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:50.427505 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 735/1482] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:51:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:51:50.433461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d20bcf-58c6-45ec-acf9-ff203e0a6714 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:50.433981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d20bcf-58c6-45ec-acf9-ff203e0a6714 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:50.434098 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d20bcf-58c6-45ec-acf9-ff203e0a6714 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:50.502449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-f3d20bcf-58c6-45ec-acf9-ff203e0a6714 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:51:50.503697 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-f3d20bcf-58c6-45ec-acf9-ff203e0a6714 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:50.550091 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:50.600426 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 748/1483] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:51:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:51:50.893751 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1732358-d7cb-4e09-9a3b-06a19b67a519 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:51:50.895317 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-a1732358-d7cb-4e09-9a3b-06a19b67a519 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:50.919377 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:51.000427 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1732358-d7cb-4e09-9a3b-06a19b67a519 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:51:51.073272 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445df15f-0259-408b-a7f5-fecb89331737 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:51:51.073881 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445df15f-0259-408b-a7f5-fecb89331737 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:51.074157 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445df15f-0259-408b-a7f5-fecb89331737 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:51.074426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445df15f-0259-408b-a7f5-fecb89331737 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:51.076059 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-445df15f-0259-408b-a7f5-fecb89331737 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:51.150409 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:51.188844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445df15f-0259-408b-a7f5-fecb89331737 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:51:51.190292 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445df15f-0259-408b-a7f5-fecb89331737 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:51:51.209285 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 736/1484] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:51:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:51:53.444773 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ff6323-b678-40a5-b8f3-89c8bf93168b None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:53.445347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ff6323-b678-40a5-b8f3-89c8bf93168b None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:53.445347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ff6323-b678-40a5-b8f3-89c8bf93168b None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:53.446711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 749/1485] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:51:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:51:53.452273 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0bcb4a-7c7e-4498-b7c1-bc46abb5edd9 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:53.452700 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0bcb4a-7c7e-4498-b7c1-bc46abb5edd9 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:53.452814 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0bcb4a-7c7e-4498-b7c1-bc46abb5edd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:53.527833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-6a0bcb4a-7c7e-4498-b7c1-bc46abb5edd9 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:51:53.529044 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-6a0bcb4a-7c7e-4498-b7c1-bc46abb5edd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:53.625976 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 737/1486] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:51:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:51:53.918202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ff6323-b678-40a5-b8f3-89c8bf93168b None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:51:53.919770 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-48ff6323-b678-40a5-b8f3-89c8bf93168b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:54.023186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ff6323-b678-40a5-b8f3-89c8bf93168b None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:51:54.099214 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:54.100713 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88312d4e-ac33-48bd-8f8c-4bce63ad5d32 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:51:54.101333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88312d4e-ac33-48bd-8f8c-4bce63ad5d32 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:54.101794 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88312d4e-ac33-48bd-8f8c-4bce63ad5d32 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:54.102230 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88312d4e-ac33-48bd-8f8c-4bce63ad5d32 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:54.104063 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-88312d4e-ac33-48bd-8f8c-4bce63ad5d32 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:54.209974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88312d4e-ac33-48bd-8f8c-4bce63ad5d32 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:51:54.211347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88312d4e-ac33-48bd-8f8c-4bce63ad5d32 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:51:54.229475 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 750/1487] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:51:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:51:56.599652 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea19db80-c059-4e7e-a370-0aa588fd8fea None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:56.600196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea19db80-c059-4e7e-a370-0aa588fd8fea None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:56.600675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea19db80-c059-4e7e-a370-0aa588fd8fea None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:56.602040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 738/1488] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:51:56 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:51:56.608279 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19207630-af49-4839-9895-7f8198f0dfe2 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:56.608910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19207630-af49-4839-9895-7f8198f0dfe2 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:56.608910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19207630-af49-4839-9895-7f8198f0dfe2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:56.687606 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-19207630-af49-4839-9895-7f8198f0dfe2 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:51:56.688846 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-19207630-af49-4839-9895-7f8198f0dfe2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:56.757738 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:56.807899 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 751/1489] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:51:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:51:57.120317 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea19db80-c059-4e7e-a370-0aa588fd8fea None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:51:57.122585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ea19db80-c059-4e7e-a370-0aa588fd8fea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:57.170733 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:57.260459 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea19db80-c059-4e7e-a370-0aa588fd8fea None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:51:57.341816 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1710f4f3-eedf-48d2-966a-e81b201ffb27 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:51:57.342435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1710f4f3-eedf-48d2-966a-e81b201ffb27 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:57.342770 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1710f4f3-eedf-48d2-966a-e81b201ffb27 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:57.343020 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1710f4f3-eedf-48d2-966a-e81b201ffb27 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:57.344737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-1710f4f3-eedf-48d2-966a-e81b201ffb27 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:51:57.436788 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:51:57.470697 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1710f4f3-eedf-48d2-966a-e81b201ffb27 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:51:57.472150 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1710f4f3-eedf-48d2-966a-e81b201ffb27 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:51:57.493224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 739/1490] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:51:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:51:59.940159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5891b69c-94fb-45c1-a186-2a99c29a403b None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:59.940724 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5891b69c-94fb-45c1-a186-2a99c29a403b None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:59.940724 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5891b69c-94fb-45c1-a186-2a99c29a403b None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:51:59.942101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 752/1491] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 18 00:51:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:51:59.947610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42054401-8a64-4cdf-88ab-34c8dce6c9ab None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:51:59.948289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42054401-8a64-4cdf-88ab-34c8dce6c9ab None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:51:59.948289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42054401-8a64-4cdf-88ab-34c8dce6c9ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:52:00.018264 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-42054401-8a64-4cdf-88ab-34c8dce6c9ab None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:52:00.019549 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-42054401-8a64-4cdf-88ab-34c8dce6c9ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:52:00.120723 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 740/1492] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 18 00:51:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:52:00.413309 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5891b69c-94fb-45c1-a186-2a99c29a403b None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:52:00.415018 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-5891b69c-94fb-45c1-a186-2a99c29a403b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:52:00.520459 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5891b69c-94fb-45c1-a186-2a99c29a403b None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:52:00.593697 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bf4ca5-4609-4cbe-a31c-295591812dde service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:52:00.594219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bf4ca5-4609-4cbe-a31c-295591812dde service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:52:00.594520 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bf4ca5-4609-4cbe-a31c-295591812dde service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:52:00.594792 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bf4ca5-4609-4cbe-a31c-295591812dde service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:52:00.596361 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c4bf4ca5-4609-4cbe-a31c-295591812dde service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:52:00.605261 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:52:00.705148 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4bf4ca5-4609-4cbe-a31c-295591812dde service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:52:00.706653 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4bf4ca5-4609-4cbe-a31c-295591812dde service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:52:00.726891 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 753/1493] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:52:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:52:10.119093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=82 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:52:10.724466 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=50 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:53:14.158872 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f7cd55-7fbe-4888-938d-b6e128ddeb44 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:53:14.160073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f7cd55-7fbe-4888-938d-b6e128ddeb44 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:53:14.160073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f7cd55-7fbe-4888-938d-b6e128ddeb44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:53:14.240451 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-c4f7cd55-7fbe-4888-938d-b6e128ddeb44 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:53:14.241683 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c4f7cd55-7fbe-4888-938d-b6e128ddeb44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:53:14.346774 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 741/1494] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:53:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:53:14.355492 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd94862-358b-4339-9321-6f49414fce06 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:53:14.355901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd94862-358b-4339-9321-6f49414fce06 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:53:14.356034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd94862-358b-4339-9321-6f49414fce06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:53:14.448400 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-efd94862-358b-4339-9321-6f49414fce06 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:53:14.449537 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-efd94862-358b-4339-9321-6f49414fce06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:53:14.550409 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 754/1495] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:53:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:53:14.555058 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f7cd55-7fbe-4888-938d-b6e128ddeb44 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:53:14.556271 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c4f7cd55-7fbe-4888-938d-b6e128ddeb44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:53:14.570731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f7cd55-7fbe-4888-938d-b6e128ddeb44 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:53:14.643521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:53:14.680855 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c11c61-58d3-471d-ba7d-c6eb756f402b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:53:14.681481 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c11c61-58d3-471d-ba7d-c6eb756f402b admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:53:14.681759 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c11c61-58d3-471d-ba7d-c6eb756f402b admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:53:14.682028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c11c61-58d3-471d-ba7d-c6eb756f402b admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:53:14.682710 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c11c61-58d3-471d-ba7d-c6eb756f402b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:53:14.683017 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c11c61-58d3-471d-ba7d-c6eb756f402b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:53:14.684034 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: WARNING py.warnings [None req-91c11c61-58d3-471d-ba7d-c6eb756f402b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 18 00:53:14.684034 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: warnings.warn(msg) Sep 18 00:53:14.684034 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: Sep 18 00:53:14.684915 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c11c61-58d3-471d-ba7d-c6eb756f402b admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:53:14.697502 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 742/1496] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:53:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:53:14.706835 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd94862-358b-4339-9321-6f49414fce06 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:53:14.716268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd94862-358b-4339-9321-6f49414fce06 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:53:14.781999 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:53:14.816529 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20d3230-394b-440f-b23f-35815506fce8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:53:14.817178 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20d3230-394b-440f-b23f-35815506fce8 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:53:14.817451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20d3230-394b-440f-b23f-35815506fce8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:53:14.817711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20d3230-394b-440f-b23f-35815506fce8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:53:14.818600 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20d3230-394b-440f-b23f-35815506fce8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:53:14.820584 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: WARNING py.warnings [None req-c20d3230-394b-440f-b23f-35815506fce8 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 18 00:53:14.820584 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: warnings.warn(msg) Sep 18 00:53:14.820584 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: Sep 18 00:53:14.821798 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20d3230-394b-440f-b23f-35815506fce8 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:53:14.880039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 755/1497] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:53:14 2022] POST /identity/v3/projects => generated 377 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:53:14.884280 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c11c61-58d3-471d-ba7d-c6eb756f402b admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:53:14.885806 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-91c11c61-58d3-471d-ba7d-c6eb756f402b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:53:14.924105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c11c61-58d3-471d-ba7d-c6eb756f402b admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:53:15.020303 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0d3bfe-5def-4474-a22c-15cdbba3b2f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:53:15.020805 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0d3bfe-5def-4474-a22c-15cdbba3b2f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:53:15.021071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0d3bfe-5def-4474-a22c-15cdbba3b2f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:53:15.021339 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0d3bfe-5def-4474-a22c-15cdbba3b2f8 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:53:15.021958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0d3bfe-5def-4474-a22c-15cdbba3b2f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:53:15.022986 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: WARNING py.warnings [None req-6e0d3bfe-5def-4474-a22c-15cdbba3b2f8 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 18 00:53:15.022986 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: warnings.warn(msg) Sep 18 00:53:15.022986 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: Sep 18 00:53:15.024187 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0d3bfe-5def-4474-a22c-15cdbba3b2f8 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:53:15.068678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 743/1498] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:53:14 2022] POST /identity/v3/users => generated 289 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:53:15.072853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20d3230-394b-440f-b23f-35815506fce8 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:53:15.074385 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c20d3230-394b-440f-b23f-35815506fce8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:53:15.140258 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20d3230-394b-440f-b23f-35815506fce8 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:53:15.222904 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:53:15.237076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e37120a-9267-4c9c-854e-d96d21e4a668 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:53:15.238198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e37120a-9267-4c9c-854e-d96d21e4a668 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:53:15.238496 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e37120a-9267-4c9c-854e-d96d21e4a668 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:53:15.238768 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e37120a-9267-4c9c-854e-d96d21e4a668 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:53:15.239409 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e37120a-9267-4c9c-854e-d96d21e4a668 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:53:15.240448 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: WARNING py.warnings [None req-5e37120a-9267-4c9c-854e-d96d21e4a668 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 18 00:53:15.240448 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: warnings.warn(msg) Sep 18 00:53:15.240448 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: Sep 18 00:53:15.241452 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e37120a-9267-4c9c-854e-d96d21e4a668 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:53:15.254564 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 756/1499] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:53:15 2022] GET /identity/v3/roles => generated 1420 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:53:15.258655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0d3bfe-5def-4474-a22c-15cdbba3b2f8 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:53:15.260163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-6e0d3bfe-5def-4474-a22c-15cdbba3b2f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:53:15.271383 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0d3bfe-5def-4474-a22c-15cdbba3b2f8 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:53:15.295423 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:53:15.366687 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb6a10d-5371-487a-901a-6c7ed7d13900 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:53:15.367287 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb6a10d-5371-487a-901a-6c7ed7d13900 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:53:15.367568 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb6a10d-5371-487a-901a-6c7ed7d13900 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:53:15.367834 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb6a10d-5371-487a-901a-6c7ed7d13900 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:53:15.368474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb6a10d-5371-487a-901a-6c7ed7d13900 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:53:15.369480 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: WARNING py.warnings [None req-5eb6a10d-5371-487a-901a-6c7ed7d13900 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 18 00:53:15.369480 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: warnings.warn(msg) Sep 18 00:53:15.369480 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: Sep 18 00:53:15.370361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb6a10d-5371-487a-901a-6c7ed7d13900 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:53:15.383499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 744/1500] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:53:15 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:53:15.387668 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e37120a-9267-4c9c-854e-d96d21e4a668 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:53:15.389184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-5e37120a-9267-4c9c-854e-d96d21e4a668 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:53:15.422241 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e37120a-9267-4c9c-854e-d96d21e4a668 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:53:15.518269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e994e8-d1f5-425e-893d-dfa0bc8e94f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:53:15.518959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e994e8-d1f5-425e-893d-dfa0bc8e94f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:53:15.519242 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e994e8-d1f5-425e-893d-dfa0bc8e94f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:53:15.519509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e994e8-d1f5-425e-893d-dfa0bc8e94f3 admin admin] PATH_INFO: `/v3/projects/8dd45672e4af45bbb95902302f69d853/users/cea3986bd9e84024a71742fd1886b2cd/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:53:15.542385 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e994e8-d1f5-425e-893d-dfa0bc8e94f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8dd45672e4af45bbb95902302f69d853, user_id=cea3986bd9e84024a71742fd1886b2cd, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:53:15.543447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: WARNING py.warnings [None req-75e994e8-d1f5-425e-893d-dfa0bc8e94f3 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 18 00:53:15.543447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: warnings.warn(msg) Sep 18 00:53:15.543447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: Sep 18 00:53:15.545381 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e994e8-d1f5-425e-893d-dfa0bc8e94f3 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:53:15.570080 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 757/1501] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:53:15 2022] PUT /identity/v3/projects/8dd45672e4af45bbb95902302f69d853/users/cea3986bd9e84024a71742fd1886b2cd/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:53:15.582194 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2f06af-92ea-4075-ac0e-dad6cd263151 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:53:15.582516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2f06af-92ea-4075-ac0e-dad6cd263151 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:53:15.582796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2f06af-92ea-4075-ac0e-dad6cd263151 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:53:15.642603 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-3e2f06af-92ea-4075-ac0e-dad6cd263151 None None] MFA Rules not processed for user `cea3986bd9e84024a71742fd1886b2cd`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:53:15.643826 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-3e2f06af-92ea-4075-ac0e-dad6cd263151 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:53:15.726211 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 745/1502] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:53:15 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:53:15.741220 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a123ec81-5738-476f-9cd0-807bfeb6b85c None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:53:15.741510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a123ec81-5738-476f-9cd0-807bfeb6b85c None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:53:15.741783 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a123ec81-5738-476f-9cd0-807bfeb6b85c None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:53:15.743147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 758/1503] 10.222.0.42 () {54 vars in 964 bytes} [Sun Sep 18 00:53:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:53:15.748069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53a989b-9112-4cb2-8b7a-6da684688046 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:53:15.748359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53a989b-9112-4cb2-8b7a-6da684688046 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:53:15.748632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53a989b-9112-4cb2-8b7a-6da684688046 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:53:15.780413 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:53:15.828458 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-b53a989b-9112-4cb2-8b7a-6da684688046 None None] MFA Rules not processed for user `c4c3eaee0a8f4de0b79841a95b35091f`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:53:15.829696 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-b53a989b-9112-4cb2-8b7a-6da684688046 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:53:15.906390 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 746/1504] 10.222.0.42 () {58 vars in 1061 bytes} [Sun Sep 18 00:53:15 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:53:15.914877 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a123ec81-5738-476f-9cd0-807bfeb6b85c None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:53:15.916634 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a123ec81-5738-476f-9cd0-807bfeb6b85c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:53:15.967582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:53:16.022650 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a123ec81-5738-476f-9cd0-807bfeb6b85c None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:53:16.096292 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1534d6e6-2a7f-4380-8dcc-18710cf59105 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:53:16.097564 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1534d6e6-2a7f-4380-8dcc-18710cf59105 service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:53:16.097844 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1534d6e6-2a7f-4380-8dcc-18710cf59105 service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:53:16.098116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1534d6e6-2a7f-4380-8dcc-18710cf59105 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:53:16.099733 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-1534d6e6-2a7f-4380-8dcc-18710cf59105 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:53:16.197177 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1534d6e6-2a7f-4380-8dcc-18710cf59105 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:53:16.198688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1534d6e6-2a7f-4380-8dcc-18710cf59105 service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:53:16.202492 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:53:16.219447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 759/1505] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:53:15 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:53:16.264297 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ee1743-2c41-4ea7-99f3-3dd5d79fa15b None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:53:16.264737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ee1743-2c41-4ea7-99f3-3dd5d79fa15b None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:53:16.264833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ee1743-2c41-4ea7-99f3-3dd5d79fa15b None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:53:16.266105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 747/1506] 10.222.0.42 () {52 vars in 830 bytes} [Sun Sep 18 00:53:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:53:16.270202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1534d6e6-2a7f-4380-8dcc-18710cf59105 service cinder] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:53:16.271442 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-1534d6e6-2a7f-4380-8dcc-18710cf59105 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:53:16.367230 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1534d6e6-2a7f-4380-8dcc-18710cf59105 service cinder] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:53:16.459614 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e95972e-d3a3-40d3-b3fd-5bfc857fc3bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:53:16.460326 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e95972e-d3a3-40d3-b3fd-5bfc857fc3bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:53:16.460543 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e95972e-d3a3-40d3-b3fd-5bfc857fc3bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:53:16.460812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e95972e-d3a3-40d3-b3fd-5bfc857fc3bf admin admin] PATH_INFO: `/v3/users/cea3986bd9e84024a71742fd1886b2cd` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:53:16.474641 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e95972e-d3a3-40d3-b3fd-5bfc857fc3bf admin admin] RBAC: Authorizing `identity:delete_user(user_id=cea3986bd9e84024a71742fd1886b2cd)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:53:16.475727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: WARNING py.warnings [None req-8e95972e-d3a3-40d3-b3fd-5bfc857fc3bf admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 18 00:53:16.475727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: warnings.warn(msg) Sep 18 00:53:16.475727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: Sep 18 00:53:16.476965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e95972e-d3a3-40d3-b3fd-5bfc857fc3bf admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:53:16.486716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:53:16.549109 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8e95972e-d3a3-40d3-b3fd-5bfc857fc3bf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cea3986bd9e84024a71742fd1886b2cd'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:53:16.558546 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8e95972e-d3a3-40d3-b3fd-5bfc857fc3bf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cea3986bd9e84024a71742fd1886b2cd'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:53:16.582651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8e95972e-d3a3-40d3-b3fd-5bfc857fc3bf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cea3986bd9e84024a71742fd1886b2cd'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:53:16.583207 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8e95972e-d3a3-40d3-b3fd-5bfc857fc3bf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cea3986bd9e84024a71742fd1886b2cd'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:53:16.596771 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8e95972e-d3a3-40d3-b3fd-5bfc857fc3bf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cea3986bd9e84024a71742fd1886b2cd'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:53:16.602189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 760/1507] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:53:16 2022] DELETE /identity/v3/users/cea3986bd9e84024a71742fd1886b2cd => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:53:16.606673 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5144e6c0-450f-4142-ace8-70b806025240 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:53:16.606900 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5144e6c0-450f-4142-ace8-70b806025240 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:53:16.607175 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5144e6c0-450f-4142-ace8-70b806025240 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:53:16.664241 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-5144e6c0-450f-4142-ace8-70b806025240 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:53:16.665445 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-5144e6c0-450f-4142-ace8-70b806025240 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:53:16.739403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:53:16.753960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 748/1508] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:53:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:53:16.765619 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e95972e-d3a3-40d3-b3fd-5bfc857fc3bf admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:53:16.766883 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-8e95972e-d3a3-40d3-b3fd-5bfc857fc3bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:53:16.866829 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e95972e-d3a3-40d3-b3fd-5bfc857fc3bf admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:53:16.918952 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:53:16.939767 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf376738-3fcc-4c13-99cb-f3f0d3502ad9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:53:16.940375 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf376738-3fcc-4c13-99cb-f3f0d3502ad9 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:53:16.941234 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf376738-3fcc-4c13-99cb-f3f0d3502ad9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:53:16.941506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf376738-3fcc-4c13-99cb-f3f0d3502ad9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:53:16.943103 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-cf376738-3fcc-4c13-99cb-f3f0d3502ad9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:53:17.052740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf376738-3fcc-4c13-99cb-f3f0d3502ad9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:53:17.054231 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf376738-3fcc-4c13-99cb-f3f0d3502ad9 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:53:17.073107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 761/1509] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:53:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:53:17.426424 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5144e6c0-450f-4142-ace8-70b806025240 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:53:17.427813 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-5144e6c0-450f-4142-ace8-70b806025240 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:53:17.439406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5144e6c0-450f-4142-ace8-70b806025240 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:53:17.543904 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f24dff-07fd-4e5d-90a9-fbf682deff85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:53:17.544534 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f24dff-07fd-4e5d-90a9-fbf682deff85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:53:17.544812 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f24dff-07fd-4e5d-90a9-fbf682deff85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:53:17.545087 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f24dff-07fd-4e5d-90a9-fbf682deff85 admin admin] PATH_INFO: `/v3/projects/8dd45672e4af45bbb95902302f69d853` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:53:17.555478 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f24dff-07fd-4e5d-90a9-fbf682deff85 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8dd45672e4af45bbb95902302f69d853)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:53:17.556498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: WARNING py.warnings [None req-48f24dff-07fd-4e5d-90a9-fbf682deff85 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 18 00:53:17.556498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: warnings.warn(msg) Sep 18 00:53:17.556498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: Sep 18 00:53:17.557689 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f24dff-07fd-4e5d-90a9-fbf682deff85 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:53:17.615656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-48f24dff-07fd-4e5d-90a9-fbf682deff85 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8dd45672e4af45bbb95902302f69d853', 'request_id': 'req-48f24dff-07fd-4e5d-90a9-fbf682deff85'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:53:17.621957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-48f24dff-07fd-4e5d-90a9-fbf682deff85 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8dd45672e4af45bbb95902302f69d853', 'request_id': 'req-48f24dff-07fd-4e5d-90a9-fbf682deff85'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:53:17.632057 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-48f24dff-07fd-4e5d-90a9-fbf682deff85 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8dd45672e4af45bbb95902302f69d853', 'request_id': 'req-48f24dff-07fd-4e5d-90a9-fbf682deff85'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:53:17.661938 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-48f24dff-07fd-4e5d-90a9-fbf682deff85 admin admin] The token cache is being invalidate because project 8dd45672e4af45bbb95902302f69d853 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:53:17.662227 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-48f24dff-07fd-4e5d-90a9-fbf682deff85 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:53:17.662779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-48f24dff-07fd-4e5d-90a9-fbf682deff85 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:53:17.664073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 749/1510] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:53:17 2022] DELETE /identity/v3/projects/8dd45672e4af45bbb95902302f69d853 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:53:27.070842 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=62 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:53:27.660130 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:54:46.517198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1af6d8-59e0-4ec5-931a-f1e990c4eabb None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:54:46.517809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1af6d8-59e0-4ec5-931a-f1e990c4eabb None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:54:46.517809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1af6d8-59e0-4ec5-931a-f1e990c4eabb None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:54:46.519129 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 762/1511] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 18 00:54:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:54:46.524313 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948f0831-73e0-46c7-841f-4618ba18f55a None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:54:46.524995 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948f0831-73e0-46c7-841f-4618ba18f55a None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:54:46.524995 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948f0831-73e0-46c7-841f-4618ba18f55a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:54:46.613773 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-948f0831-73e0-46c7-841f-4618ba18f55a None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:54:46.615037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-948f0831-73e0-46c7-841f-4618ba18f55a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:54:46.718188 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 750/1512] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 18 00:54:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:54:46.875994 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1af6d8-59e0-4ec5-931a-f1e990c4eabb None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:54:46.878258 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-af1af6d8-59e0-4ec5-931a-f1e990c4eabb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:54:47.002961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1af6d8-59e0-4ec5-931a-f1e990c4eabb None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:54:47.008505 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8948107d-85ab-45ee-ac42-f93d96ccb93d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:54:47.009145 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8948107d-85ab-45ee-ac42-f93d96ccb93d service nova] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:54:47.009424 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8948107d-85ab-45ee-ac42-f93d96ccb93d service nova] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:54:47.009693 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8948107d-85ab-45ee-ac42-f93d96ccb93d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:54:47.018851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8948107d-85ab-45ee-ac42-f93d96ccb93d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:54:47.020490 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8948107d-85ab-45ee-ac42-f93d96ccb93d service nova] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:54:47.030406 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 763/1513] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:54:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:54:47.709550 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46e9e8e-94c0-452a-8eaa-35cfaca0408e None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:54:47.710696 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46e9e8e-94c0-452a-8eaa-35cfaca0408e None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:54:47.710696 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46e9e8e-94c0-452a-8eaa-35cfaca0408e None None] PATH_INFO: `/v3` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:54:47.712120 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 751/1514] 10.222.0.49 () {54 vars in 925 bytes} [Sun Sep 18 00:54:47 2022] GET /identity/v3 => generated 255 bytes in 4 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:54:47.730717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f73e7f2-c126-4c24-a2e1-94b1e0d69393 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:54:47.731218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f73e7f2-c126-4c24-a2e1-94b1e0d69393 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:54:47.731319 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f73e7f2-c126-4c24-a2e1-94b1e0d69393 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:54:47.797870 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-0f73e7f2-c126-4c24-a2e1-94b1e0d69393 None None] MFA Rules not processed for user `8f30116fb5ce43dab69efa8bafd9d049`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:54:47.799010 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-0f73e7f2-c126-4c24-a2e1-94b1e0d69393 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:54:47.872744 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:54:47.878966 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 764/1515] 10.222.0.49 () {58 vars in 1015 bytes} [Sun Sep 18 00:54:47 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:54:48.130378 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d9f896-36f2-43e7-bf9e-0b3e6f0da056 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:54:48.130707 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d9f896-36f2-43e7-bf9e-0b3e6f0da056 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:54:48.130990 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d9f896-36f2-43e7-bf9e-0b3e6f0da056 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:54:48.132339 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 752/1516] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 18 00:54:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:54:48.142102 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a89da6-d5aa-4e83-91ae-ffb305fedd82 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:54:48.142537 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a89da6-d5aa-4e83-91ae-ffb305fedd82 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:54:48.142861 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a89da6-d5aa-4e83-91ae-ffb305fedd82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:54:48.165160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-33a89da6-d5aa-4e83-91ae-ffb305fedd82 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:54:48.166377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-33a89da6-d5aa-4e83-91ae-ffb305fedd82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:54:48.174261 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 765/1517] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 18 00:54:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:54:48.325626 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d9f896-36f2-43e7-bf9e-0b3e6f0da056 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:54:48.341209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d9f896-36f2-43e7-bf9e-0b3e6f0da056 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:54:48.346717 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f0c9c1-dbf7-43e0-aa70-edb4dcdea5d5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:54:48.347316 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f0c9c1-dbf7-43e0-aa70-edb4dcdea5d5 service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:54:48.347592 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f0c9c1-dbf7-43e0-aa70-edb4dcdea5d5 service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:54:48.347861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f0c9c1-dbf7-43e0-aa70-edb4dcdea5d5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:54:48.356484 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f0c9c1-dbf7-43e0-aa70-edb4dcdea5d5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:54:48.358187 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f0c9c1-dbf7-43e0-aa70-edb4dcdea5d5 service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:54:48.367678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 753/1518] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:54:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:54:56.710155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a282f5-b1ad-48e9-9dbb-82ba1d8875ee None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:54:56.710654 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a282f5-b1ad-48e9-9dbb-82ba1d8875ee None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:54:56.710772 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a282f5-b1ad-48e9-9dbb-82ba1d8875ee None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:54:56.712130 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 766/1519] 10.222.0.42 () {54 vars in 961 bytes} [Sun Sep 18 00:54:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:54:56.717324 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922c8687-a921-4e66-8a9a-48c10b067beb None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:54:56.717792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922c8687-a921-4e66-8a9a-48c10b067beb None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:54:56.717899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922c8687-a921-4e66-8a9a-48c10b067beb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:54:56.738588 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-922c8687-a921-4e66-8a9a-48c10b067beb None None] MFA Rules not processed for user `8f30116fb5ce43dab69efa8bafd9d049`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:54:56.739736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-922c8687-a921-4e66-8a9a-48c10b067beb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:54:56.748911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 754/1520] 10.222.0.42 () {58 vars in 1058 bytes} [Sun Sep 18 00:54:56 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:54:56.757061 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a282f5-b1ad-48e9-9dbb-82ba1d8875ee None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:54:56.770798 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a282f5-b1ad-48e9-9dbb-82ba1d8875ee None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:54:56.775774 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8407dec3-e50b-4985-bbfb-289698aab604 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8f30116fb5ce43dab69efa8bafd9d049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:54:56.776371 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8407dec3-e50b-4985-bbfb-289698aab604 service placement] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:54:56.776644 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8407dec3-e50b-4985-bbfb-289698aab604 service placement] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:54:56.776915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8407dec3-e50b-4985-bbfb-289698aab604 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:54:56.785541 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8407dec3-e50b-4985-bbfb-289698aab604 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:54:56.787157 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8407dec3-e50b-4985-bbfb-289698aab604 service placement] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:54:56.797669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 767/1521] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:54:56 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:54:59.848403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8afe305-4f0a-4c61-b6b3-4c4c84a91929 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:54:59.848976 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8afe305-4f0a-4c61-b6b3-4c4c84a91929 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:54:59.848976 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8afe305-4f0a-4c61-b6b3-4c4c84a91929 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:54:59.850271 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 755/1522] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 18 00:54:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:54:59.856901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ceecf8-8970-4bcb-971c-b1667426f99d None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:54:59.857351 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ceecf8-8970-4bcb-971c-b1667426f99d None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:54:59.857440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ceecf8-8970-4bcb-971c-b1667426f99d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:54:59.876559 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-66ceecf8-8970-4bcb-971c-b1667426f99d None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:54:59.877758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-66ceecf8-8970-4bcb-971c-b1667426f99d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:54:59.886044 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 768/1523] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 18 00:54:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:55:00.037945 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8afe305-4f0a-4c61-b6b3-4c4c84a91929 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:55:00.052554 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8afe305-4f0a-4c61-b6b3-4c4c84a91929 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:55:00.058419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09b0432-d6d7-42d5-98a2-45cbee240789 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:55:00.059077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09b0432-d6d7-42d5-98a2-45cbee240789 service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:55:00.059351 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09b0432-d6d7-42d5-98a2-45cbee240789 service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:55:00.059616 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09b0432-d6d7-42d5-98a2-45cbee240789 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:55:00.068168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09b0432-d6d7-42d5-98a2-45cbee240789 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:55:00.071465 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09b0432-d6d7-42d5-98a2-45cbee240789 service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:55:00.081036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 756/1524] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:55:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:55:01.173536 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4531ab-5c57-4fca-9683-c7a63fde5d0d None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:55:01.174125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4531ab-5c57-4fca-9683-c7a63fde5d0d None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:55:01.174125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4531ab-5c57-4fca-9683-c7a63fde5d0d None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:55:01.175476 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 769/1525] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 18 00:55:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:55:01.182565 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93e3b79-ab5b-475d-bd59-16c3d5975123 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:55:01.183341 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93e3b79-ab5b-475d-bd59-16c3d5975123 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:55:01.183341 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93e3b79-ab5b-475d-bd59-16c3d5975123 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:55:01.203589 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-d93e3b79-ab5b-475d-bd59-16c3d5975123 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:55:01.204799 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d93e3b79-ab5b-475d-bd59-16c3d5975123 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:55:01.213548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 757/1526] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 18 00:55:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:55:01.366272 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4531ab-5c57-4fca-9683-c7a63fde5d0d None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:55:01.380523 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4531ab-5c57-4fca-9683-c7a63fde5d0d None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:55:01.386356 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895cb42d-29c2-4f82-9126-60360f72e58d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:55:01.386981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895cb42d-29c2-4f82-9126-60360f72e58d service nova] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:55:01.387259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895cb42d-29c2-4f82-9126-60360f72e58d service nova] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:55:01.387529 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895cb42d-29c2-4f82-9126-60360f72e58d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:55:01.395607 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895cb42d-29c2-4f82-9126-60360f72e58d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:55:01.397189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895cb42d-29c2-4f82-9126-60360f72e58d service nova] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:55:01.407581 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 770/1527] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:55:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:55:05.590253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f419b58-dcab-49fe-bd7f-3d1e2dba9829 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:55:05.590859 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f419b58-dcab-49fe-bd7f-3d1e2dba9829 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:55:05.590859 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f419b58-dcab-49fe-bd7f-3d1e2dba9829 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:55:05.592184 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 758/1528] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 18 00:55:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:55:05.597544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df957895-c6fc-4951-92ef-da19e692f92b None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:55:05.597990 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df957895-c6fc-4951-92ef-da19e692f92b None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:55:05.598095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df957895-c6fc-4951-92ef-da19e692f92b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:55:05.618292 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-df957895-c6fc-4951-92ef-da19e692f92b None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:55:05.619508 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-df957895-c6fc-4951-92ef-da19e692f92b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:55:05.627260 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 771/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 18 00:55:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:55:05.926674 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f419b58-dcab-49fe-bd7f-3d1e2dba9829 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:55:05.928404 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-0f419b58-dcab-49fe-bd7f-3d1e2dba9829 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:55:05.962115 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:55:06.011627 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f419b58-dcab-49fe-bd7f-3d1e2dba9829 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:55:06.015200 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93da11f8-e33d-4c78-acc0-4e219043008e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:55:06.015802 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93da11f8-e33d-4c78-acc0-4e219043008e service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:55:06.016080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93da11f8-e33d-4c78-acc0-4e219043008e service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:55:06.016353 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93da11f8-e33d-4c78-acc0-4e219043008e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:55:06.025846 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93da11f8-e33d-4c78-acc0-4e219043008e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:55:06.027303 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93da11f8-e33d-4c78-acc0-4e219043008e service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:55:06.036572 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 759/1530] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:55:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:55:08.277872 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1b3fbe-2843-48a7-9bf3-5dd4fe5cd87c None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:55:08.278447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1b3fbe-2843-48a7-9bf3-5dd4fe5cd87c None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:55:08.278447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1b3fbe-2843-48a7-9bf3-5dd4fe5cd87c None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:55:08.279825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 772/1531] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 18 00:55:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:55:08.285135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3016a2-c4b2-44f7-a507-d324d0f0e078 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:55:08.285540 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3016a2-c4b2-44f7-a507-d324d0f0e078 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:55:08.285686 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3016a2-c4b2-44f7-a507-d324d0f0e078 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:55:08.305665 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-1b3016a2-c4b2-44f7-a507-d324d0f0e078 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:55:08.306904 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-1b3016a2-c4b2-44f7-a507-d324d0f0e078 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:55:08.314729 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 760/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 18 00:55:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:55:08.602495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1b3fbe-2843-48a7-9bf3-5dd4fe5cd87c None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:55:08.604166 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-ef1b3fbe-2843-48a7-9bf3-5dd4fe5cd87c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:55:08.617386 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1b3fbe-2843-48a7-9bf3-5dd4fe5cd87c None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:55:08.620881 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683b379e-3775-49dc-979e-71cd00341d3b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:55:08.621467 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683b379e-3775-49dc-979e-71cd00341d3b service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:55:08.621739 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683b379e-3775-49dc-979e-71cd00341d3b service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:55:08.622007 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683b379e-3775-49dc-979e-71cd00341d3b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:55:08.630590 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683b379e-3775-49dc-979e-71cd00341d3b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:55:08.632041 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683b379e-3775-49dc-979e-71cd00341d3b service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:55:08.641061 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 773/1533] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:55:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:55:12.552787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb75d3d-b40b-4d47-ac93-53bc62c339d3 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:55:12.553338 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb75d3d-b40b-4d47-ac93-53bc62c339d3 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:55:12.553338 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb75d3d-b40b-4d47-ac93-53bc62c339d3 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:55:12.554929 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 761/1534] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 18 00:55:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:55:12.560943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a90e27f-7171-43f8-98ab-22fa98f27f8b None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:55:12.561671 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a90e27f-7171-43f8-98ab-22fa98f27f8b None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:55:12.561953 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a90e27f-7171-43f8-98ab-22fa98f27f8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:55:12.582990 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-0a90e27f-7171-43f8-98ab-22fa98f27f8b None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:55:12.584173 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-0a90e27f-7171-43f8-98ab-22fa98f27f8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:55:12.592458 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 774/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 18 00:55:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:55:12.896307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb75d3d-b40b-4d47-ac93-53bc62c339d3 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:55:12.897994 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-edb75d3d-b40b-4d47-ac93-53bc62c339d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:55:12.975198 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb75d3d-b40b-4d47-ac93-53bc62c339d3 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:55:12.978918 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a8c35b-a782-40a4-9c14-4bfdecbec3c0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:55:12.979530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a8c35b-a782-40a4-9c14-4bfdecbec3c0 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:55:12.979810 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a8c35b-a782-40a4-9c14-4bfdecbec3c0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:55:12.980084 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a8c35b-a782-40a4-9c14-4bfdecbec3c0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:55:12.988972 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a8c35b-a782-40a4-9c14-4bfdecbec3c0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:55:12.990406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a8c35b-a782-40a4-9c14-4bfdecbec3c0 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:55:13.000279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 762/1536] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:55:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:55:15.145880 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522c014a-1abe-4019-9b19-d6aca3d7ca77 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:55:15.146419 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522c014a-1abe-4019-9b19-d6aca3d7ca77 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:55:15.146419 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522c014a-1abe-4019-9b19-d6aca3d7ca77 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:55:15.147817 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 775/1537] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 18 00:55:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:55:15.153106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344ffe8f-16b4-4890-950c-bfd8d3ccf157 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:55:15.153513 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344ffe8f-16b4-4890-950c-bfd8d3ccf157 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:55:15.153668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344ffe8f-16b4-4890-950c-bfd8d3ccf157 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:55:15.173666 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-344ffe8f-16b4-4890-950c-bfd8d3ccf157 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:55:15.174894 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-344ffe8f-16b4-4890-950c-bfd8d3ccf157 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:55:15.183190 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 763/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 18 00:55:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:55:15.481736 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522c014a-1abe-4019-9b19-d6aca3d7ca77 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:55:15.483517 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-522c014a-1abe-4019-9b19-d6aca3d7ca77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:55:15.497276 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522c014a-1abe-4019-9b19-d6aca3d7ca77 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:55:15.500912 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16d6aa7-28b0-48b1-9699-cc6319b68a2d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:55:15.501510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16d6aa7-28b0-48b1-9699-cc6319b68a2d service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:55:15.501816 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16d6aa7-28b0-48b1-9699-cc6319b68a2d service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:55:15.502082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16d6aa7-28b0-48b1-9699-cc6319b68a2d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:55:15.511648 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16d6aa7-28b0-48b1-9699-cc6319b68a2d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:55:15.513111 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16d6aa7-28b0-48b1-9699-cc6319b68a2d service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:55:15.525620 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 776/1539] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:55:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:55:25.167298 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=63 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:55:25.518612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=37 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:44.054075 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42c481d-0ef5-4ee6-a457-3175f64d1ea6 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:44.054075 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42c481d-0ef5-4ee6-a457-3175f64d1ea6 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:44.054075 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42c481d-0ef5-4ee6-a457-3175f64d1ea6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:44.054075 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-b42c481d-0ef5-4ee6-a457-3175f64d1ea6 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:44.054075 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-b42c481d-0ef5-4ee6-a457-3175f64d1ea6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:44.054075 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 764/1540] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:56:44.054948 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0f03ca-db53-41c5-8c5e-fb4c139b97bc None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:44.054948 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0f03ca-db53-41c5-8c5e-fb4c139b97bc None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:44.054948 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0f03ca-db53-41c5-8c5e-fb4c139b97bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:44.083694 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-3a0f03ca-db53-41c5-8c5e-fb4c139b97bc None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:44.085298 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3a0f03ca-db53-41c5-8c5e-fb4c139b97bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:44.104142 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 777/1541] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:44.109989 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42c481d-0ef5-4ee6-a457-3175f64d1ea6 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:44.138754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42c481d-0ef5-4ee6-a457-3175f64d1ea6 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:44.155652 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4890f2b3-9340-4933-b64d-2ec23ef9d134 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:44.157263 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4890f2b3-9340-4933-b64d-2ec23ef9d134 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:44.158280 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4890f2b3-9340-4933-b64d-2ec23ef9d134 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:44.159133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4890f2b3-9340-4933-b64d-2ec23ef9d134 admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:44.160712 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4890f2b3-9340-4933-b64d-2ec23ef9d134 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:56:44.161580 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4890f2b3-9340-4933-b64d-2ec23ef9d134 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:44.165914 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4890f2b3-9340-4933-b64d-2ec23ef9d134 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:44.200364 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 765/1542] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:56:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:56:44.208309 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0f03ca-db53-41c5-8c5e-fb4c139b97bc None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:44.223631 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0f03ca-db53-41c5-8c5e-fb4c139b97bc None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:44.238329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a03ab1c-8a4e-4409-8229-c7e827d34c89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:44.239007 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a03ab1c-8a4e-4409-8229-c7e827d34c89 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:44.239305 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a03ab1c-8a4e-4409-8229-c7e827d34c89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:44.239614 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a03ab1c-8a4e-4409-8229-c7e827d34c89 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:44.241159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a03ab1c-8a4e-4409-8229-c7e827d34c89 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:44.243363 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a03ab1c-8a4e-4409-8229-c7e827d34c89 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:44.301937 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9c2126-3d35-4f78-a3c7-6404fc771d57 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:44.303635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9c2126-3d35-4f78-a3c7-6404fc771d57 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:44.304420 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9c2126-3d35-4f78-a3c7-6404fc771d57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:44.315245 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 778/1543] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:56:44 2022] POST /identity/v3/projects => generated 371 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:44.324915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a03ab1c-8a4e-4409-8229-c7e827d34c89 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:44.368891 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-3d9c2126-3d35-4f78-a3c7-6404fc771d57 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:44.371588 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-3d9c2126-3d35-4f78-a3c7-6404fc771d57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:44.438196 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a03ab1c-8a4e-4409-8229-c7e827d34c89 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:44.452514 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0635e7-b700-4357-aa17-c5875ece0f9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:44.453488 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0635e7-b700-4357-aa17-c5875ece0f9c admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:44.453790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0635e7-b700-4357-aa17-c5875ece0f9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:44.454052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0635e7-b700-4357-aa17-c5875ece0f9c admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:44.454726 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0635e7-b700-4357-aa17-c5875ece0f9c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:44.455859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: WARNING py.warnings [None req-bc0635e7-b700-4357-aa17-c5875ece0f9c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 18 00:56:44.455859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: warnings.warn(msg) Sep 18 00:56:44.455859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: Sep 18 00:56:44.457363 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0635e7-b700-4357-aa17-c5875ece0f9c admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:44.547779 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 779/1544] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:56:44 2022] POST /identity/v3/users => generated 286 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:56:44.549819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ed5b6c-10bc-436b-bfb8-afecece91926 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:44.550302 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ed5b6c-10bc-436b-bfb8-afecece91926 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:44.557477 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ed5b6c-10bc-436b-bfb8-afecece91926 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:44.561973 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 766/1545] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 262 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:44.566509 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1c79db-2089-4c6c-a294-11f8af838a85 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:44.566509 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1c79db-2089-4c6c-a294-11f8af838a85 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:44.566509 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1c79db-2089-4c6c-a294-11f8af838a85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:44.618494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-eb1c79db-2089-4c6c-a294-11f8af838a85 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:44.618494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-eb1c79db-2089-4c6c-a294-11f8af838a85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:44.618936 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-f7ed5b6c-10bc-436b-bfb8-afecece91926 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:44.619662 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-f7ed5b6c-10bc-436b-bfb8-afecece91926 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:44.629734 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 767/1546] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:44.632336 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1c79db-2089-4c6c-a294-11f8af838a85 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:44.642178 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 780/1547] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:44.643587 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01cffe9-5882-47c3-8fb7-b5ccb1565847 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:44.643954 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01cffe9-5882-47c3-8fb7-b5ccb1565847 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:44.644218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01cffe9-5882-47c3-8fb7-b5ccb1565847 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:44.654154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1c79db-2089-4c6c-a294-11f8af838a85 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:44.668928 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70dfd0bc-a038-4f56-b3a4-b87c6e701bb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:44.669564 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dfd0bc-a038-4f56-b3a4-b87c6e701bb3 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:44.669872 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dfd0bc-a038-4f56-b3a4-b87c6e701bb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:44.670152 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dfd0bc-a038-4f56-b3a4-b87c6e701bb3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:44.674750 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70dfd0bc-a038-4f56-b3a4-b87c6e701bb3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:44.676434 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70dfd0bc-a038-4f56-b3a4-b87c6e701bb3 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:44.690138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-f01cffe9-5882-47c3-8fb7-b5ccb1565847 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:44.696894 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-f01cffe9-5882-47c3-8fb7-b5ccb1565847 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:44.708424 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 768/1548] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:44 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:44.712333 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1aabb95-0c65-4ee4-9fe5-8bb4c0c037ed None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:44.714555 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1aabb95-0c65-4ee4-9fe5-8bb4c0c037ed None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:44.717572 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1aabb95-0c65-4ee4-9fe5-8bb4c0c037ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:44.726822 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 781/1549] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:44.728308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29dbe70f-f4ec-4ef0-8706-fc04262a901b None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:44.728601 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29dbe70f-f4ec-4ef0-8706-fc04262a901b None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:44.728858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29dbe70f-f4ec-4ef0-8706-fc04262a901b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:44.777831 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-a1aabb95-0c65-4ee4-9fe5-8bb4c0c037ed None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:44.779937 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-a1aabb95-0c65-4ee4-9fe5-8bb4c0c037ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:44.782305 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-29dbe70f-f4ec-4ef0-8706-fc04262a901b None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:44.784167 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-29dbe70f-f4ec-4ef0-8706-fc04262a901b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:44.794528 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 782/1550] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:44.795443 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 769/1551] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:44.795884 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4eb34b4-6248-455e-b551-547fc3da9dd0 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:44.796988 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1aabb95-0c65-4ee4-9fe5-8bb4c0c037ed None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:44.798057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4eb34b4-6248-455e-b551-547fc3da9dd0 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:44.803256 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4eb34b4-6248-455e-b551-547fc3da9dd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:44.816153 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1aabb95-0c65-4ee4-9fe5-8bb4c0c037ed None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:44.823626 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cf950d-9c24-45a8-a6dd-924ef658b382 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:44.824889 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cf950d-9c24-45a8-a6dd-924ef658b382 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:44.825264 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cf950d-9c24-45a8-a6dd-924ef658b382 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:44.825547 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cf950d-9c24-45a8-a6dd-924ef658b382 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:44.826200 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cf950d-9c24-45a8-a6dd-924ef658b382 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:44.828319 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cf950d-9c24-45a8-a6dd-924ef658b382 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:44.842846 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 770/1552] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:44 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:44.843823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cf950d-9c24-45a8-a6dd-924ef658b382 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:44.844085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-b4eb34b4-6248-455e-b551-547fc3da9dd0 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:44.846382 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-b4eb34b4-6248-455e-b551-547fc3da9dd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:44.856442 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cf950d-9c24-45a8-a6dd-924ef658b382 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:44.859707 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 783/1553] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:44.861569 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff7a8de-d431-4bf1-90ed-86dad91b4cfa None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:44.862006 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff7a8de-d431-4bf1-90ed-86dad91b4cfa None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:44.862414 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff7a8de-d431-4bf1-90ed-86dad91b4cfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:44.866494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999ffec3-3ede-4125-895a-a7d4106dfec0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:44.867146 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999ffec3-3ede-4125-895a-a7d4106dfec0 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:44.867443 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999ffec3-3ede-4125-895a-a7d4106dfec0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:44.867997 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999ffec3-3ede-4125-895a-a7d4106dfec0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:44.868682 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999ffec3-3ede-4125-895a-a7d4106dfec0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:56:44.868987 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999ffec3-3ede-4125-895a-a7d4106dfec0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:44.870682 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999ffec3-3ede-4125-895a-a7d4106dfec0 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:44.889088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 771/1554] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:56:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:56:44.890552 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999ffec3-3ede-4125-895a-a7d4106dfec0 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:44.898235 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-bff7a8de-d431-4bf1-90ed-86dad91b4cfa None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:44.900622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-bff7a8de-d431-4bf1-90ed-86dad91b4cfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:44.915320 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999ffec3-3ede-4125-895a-a7d4106dfec0 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:44.915699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 784/1555] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:44.922524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd2be55-aca3-48f6-9922-77f6f629fd25 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:44.922524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd2be55-aca3-48f6-9922-77f6f629fd25 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:44.922524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd2be55-aca3-48f6-9922-77f6f629fd25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:44.931225 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2536cf-097c-4aee-af02-f536ee1e21a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:44.932294 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2536cf-097c-4aee-af02-f536ee1e21a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:44.932746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2536cf-097c-4aee-af02-f536ee1e21a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:44.933140 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2536cf-097c-4aee-af02-f536ee1e21a8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:44.934201 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2536cf-097c-4aee-af02-f536ee1e21a8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:56:44.934774 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2536cf-097c-4aee-af02-f536ee1e21a8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:44.937606 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2536cf-097c-4aee-af02-f536ee1e21a8 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:44.952158 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-3bd2be55-aca3-48f6-9922-77f6f629fd25 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:44.953830 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3bd2be55-aca3-48f6-9922-77f6f629fd25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:44.959806 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 772/1556] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:56:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:56:44.961192 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2536cf-097c-4aee-af02-f536ee1e21a8 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:44.969648 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 785/1557] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:44.969648 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126dbca0-08d3-4307-828b-9272e175a993 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:44.969949 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126dbca0-08d3-4307-828b-9272e175a993 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:44.970396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126dbca0-08d3-4307-828b-9272e175a993 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:44.977662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2536cf-097c-4aee-af02-f536ee1e21a8 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:44.982772 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583427d9-264f-4995-80ce-650e48cdf4c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:44.983512 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583427d9-264f-4995-80ce-650e48cdf4c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:44.983934 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583427d9-264f-4995-80ce-650e48cdf4c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:44.984270 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583427d9-264f-4995-80ce-650e48cdf4c2 admin admin] PATH_INFO: `/v3/projects/ee923b3faeca4113af384cc85257b516/users/30aa97b3fe804a90b252b3046d3ff401/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:45.000397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-126dbca0-08d3-4307-828b-9272e175a993 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:45.001641 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-126dbca0-08d3-4307-828b-9272e175a993 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:45.007078 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583427d9-264f-4995-80ce-650e48cdf4c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee923b3faeca4113af384cc85257b516, user_id=30aa97b3fe804a90b252b3046d3ff401, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:45.008942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: WARNING py.warnings [None req-583427d9-264f-4995-80ce-650e48cdf4c2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 18 00:56:45.008942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: warnings.warn(msg) Sep 18 00:56:45.008942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: Sep 18 00:56:45.010344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 786/1558] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:45.011498 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126dbca0-08d3-4307-828b-9272e175a993 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:45.016450 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583427d9-264f-4995-80ce-650e48cdf4c2 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:45.027933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126dbca0-08d3-4307-828b-9272e175a993 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:45.035519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 773/1559] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:56:44 2022] PUT /identity/v3/projects/ee923b3faeca4113af384cc85257b516/users/30aa97b3fe804a90b252b3046d3ff401/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:56:45.036748 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583427d9-264f-4995-80ce-650e48cdf4c2 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:45.112733 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd062c86-3a99-4213-b0a1-64a26833f6b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:45.113647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd062c86-3a99-4213-b0a1-64a26833f6b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:45.113647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd062c86-3a99-4213-b0a1-64a26833f6b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:45.113964 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd062c86-3a99-4213-b0a1-64a26833f6b5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:45.114599 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd062c86-3a99-4213-b0a1-64a26833f6b5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:56:45.114901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd062c86-3a99-4213-b0a1-64a26833f6b5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:45.115972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: WARNING py.warnings [None req-fd062c86-3a99-4213-b0a1-64a26833f6b5 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 18 00:56:45.115972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: warnings.warn(msg) Sep 18 00:56:45.115972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: Sep 18 00:56:45.116960 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd062c86-3a99-4213-b0a1-64a26833f6b5 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:45.127032 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:45.129093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:45.142331 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 787/1560] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:56:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:56:45.142331 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee8a1b0-e318-4cb7-b94c-98d690f914f3 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:45.142331 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee8a1b0-e318-4cb7-b94c-98d690f914f3 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:45.142331 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee8a1b0-e318-4cb7-b94c-98d690f914f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:45.143101 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583427d9-264f-4995-80ce-650e48cdf4c2 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:45.159262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-aee8a1b0-e318-4cb7-b94c-98d690f914f3 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:45.160496 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-aee8a1b0-e318-4cb7-b94c-98d690f914f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:45.160846 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90e7032-e92c-4125-9ad7-33f96f41b637 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:45.161594 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90e7032-e92c-4125-9ad7-33f96f41b637 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:45.162020 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90e7032-e92c-4125-9ad7-33f96f41b637 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:45.162444 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90e7032-e92c-4125-9ad7-33f96f41b637 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:45.163304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90e7032-e92c-4125-9ad7-33f96f41b637 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:45.164578 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: WARNING py.warnings [None req-e90e7032-e92c-4125-9ad7-33f96f41b637 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 18 00:56:45.164578 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: warnings.warn(msg) Sep 18 00:56:45.164578 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: Sep 18 00:56:45.166359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90e7032-e92c-4125-9ad7-33f96f41b637 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:45.169653 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 788/1561] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:45.171107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c158a56a-1a7a-4934-aa6c-3129fd8473cc None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:45.171492 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c158a56a-1a7a-4934-aa6c-3129fd8473cc None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:45.171867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c158a56a-1a7a-4934-aa6c-3129fd8473cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:45.196593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-c158a56a-1a7a-4934-aa6c-3129fd8473cc None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:45.199014 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c158a56a-1a7a-4934-aa6c-3129fd8473cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:45.214188 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 774/1562] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:56:45 2022] POST /identity/v3/projects => generated 379 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:45.214188 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90e7032-e92c-4125-9ad7-33f96f41b637 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:45.293772 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90e7032-e92c-4125-9ad7-33f96f41b637 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:45.298898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fea4a8-d656-4851-b63b-4b5ea3e3c751 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:45.299511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fea4a8-d656-4851-b63b-4b5ea3e3c751 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:45.299792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fea4a8-d656-4851-b63b-4b5ea3e3c751 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:45.300061 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fea4a8-d656-4851-b63b-4b5ea3e3c751 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:45.300704 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5fea4a8-d656-4851-b63b-4b5ea3e3c751 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:45.302698 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5fea4a8-d656-4851-b63b-4b5ea3e3c751 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:45.311040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 789/1563] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:45.312424 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c158a56a-1a7a-4934-aa6c-3129fd8473cc None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:45.330642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c158a56a-1a7a-4934-aa6c-3129fd8473cc None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:45.336860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fd464a-41be-4800-a002-a05db5c222fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:45.337588 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fd464a-41be-4800-a002-a05db5c222fd admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:45.337934 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fd464a-41be-4800-a002-a05db5c222fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:45.338332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fd464a-41be-4800-a002-a05db5c222fd admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:45.339227 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55fd464a-41be-4800-a002-a05db5c222fd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:56:45.339619 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55fd464a-41be-4800-a002-a05db5c222fd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:45.341577 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55fd464a-41be-4800-a002-a05db5c222fd admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:45.341801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 775/1564] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:56:45 2022] POST /identity/v3/projects => generated 389 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:45.343718 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5336395f-b580-46d3-94b5-1030df2945d0 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:45.344429 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5336395f-b580-46d3-94b5-1030df2945d0 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:45.344620 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5336395f-b580-46d3-94b5-1030df2945d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:45.370543 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 790/1565] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:56:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:56:45.370543 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0158653f-b406-42f9-90b5-b8b14350ab53 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:45.370543 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0158653f-b406-42f9-90b5-b8b14350ab53 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:45.370543 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0158653f-b406-42f9-90b5-b8b14350ab53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:45.376035 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-5336395f-b580-46d3-94b5-1030df2945d0 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:45.377599 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-5336395f-b580-46d3-94b5-1030df2945d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:45.404746 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-0158653f-b406-42f9-90b5-b8b14350ab53 None None] MFA Rules not processed for user `30aa97b3fe804a90b252b3046d3ff401`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:45.406170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-0158653f-b406-42f9-90b5-b8b14350ab53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:45.455830 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 776/1566] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:45.457265 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5336395f-b580-46d3-94b5-1030df2945d0 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:45.472426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5336395f-b580-46d3-94b5-1030df2945d0 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:45.478819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46033cd-61f3-48c7-9757-6d327068897b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:45.479542 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46033cd-61f3-48c7-9757-6d327068897b admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:45.479891 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46033cd-61f3-48c7-9757-6d327068897b admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:45.480220 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46033cd-61f3-48c7-9757-6d327068897b admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:45.480996 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46033cd-61f3-48c7-9757-6d327068897b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:45.483037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46033cd-61f3-48c7-9757-6d327068897b admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:45.491032 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 791/1567] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:45 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:45.492074 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0158653f-b406-42f9-90b5-b8b14350ab53 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:45.517579 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0158653f-b406-42f9-90b5-b8b14350ab53 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:45.535294 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cf2fc2-43b6-41ac-bfd1-b6d5e5b99fe9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:45.537829 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 777/1568] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:56:45 2022] POST /identity/v3/projects => generated 405 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:45.539338 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cf2fc2-43b6-41ac-bfd1-b6d5e5b99fe9 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:45.539976 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cf2fc2-43b6-41ac-bfd1-b6d5e5b99fe9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:45.541617 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46033cd-61f3-48c7-9757-6d327068897b admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:45.543366 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cf2fc2-43b6-41ac-bfd1-b6d5e5b99fe9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:45.543366 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95cf2fc2-43b6-41ac-bfd1-b6d5e5b99fe9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:56:45.543366 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95cf2fc2-43b6-41ac-bfd1-b6d5e5b99fe9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:45.545269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95cf2fc2-43b6-41ac-bfd1-b6d5e5b99fe9 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:45.562087 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 792/1569] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:56:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:56:45.563601 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e481305-ff5c-4fb9-9688-2542532d212f None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:45.563882 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e481305-ff5c-4fb9-9688-2542532d212f None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:45.564157 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e481305-ff5c-4fb9-9688-2542532d212f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:45.587856 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-9e481305-ff5c-4fb9-9688-2542532d212f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:45.589107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-9e481305-ff5c-4fb9-9688-2542532d212f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:45.628987 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:45.651658 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46033cd-61f3-48c7-9757-6d327068897b admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:45.658287 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c31388-892e-4685-8438-b435359db182 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:45.659277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c31388-892e-4685-8438-b435359db182 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:45.659720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c31388-892e-4685-8438-b435359db182 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:45.660163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c31388-892e-4685-8438-b435359db182 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:45.661195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c31388-892e-4685-8438-b435359db182 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:45.661361 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:45.665341 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c31388-892e-4685-8438-b435359db182 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:45.678289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 793/1570] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:45.679809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e481305-ff5c-4fb9-9688-2542532d212f None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:45.695608 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e481305-ff5c-4fb9-9688-2542532d212f None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:45.702109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 778/1571] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:56:45 2022] POST /identity/v3/users => generated 290 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:45.703469 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c31388-892e-4685-8438-b435359db182 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:45.706685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc1ce56-7f1f-40b4-8334-7068e7d96802 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:45.707492 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc1ce56-7f1f-40b4-8334-7068e7d96802 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:45.707873 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc1ce56-7f1f-40b4-8334-7068e7d96802 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:45.708251 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc1ce56-7f1f-40b4-8334-7068e7d96802 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:45.712451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc1ce56-7f1f-40b4-8334-7068e7d96802 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:45.713146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc1ce56-7f1f-40b4-8334-7068e7d96802 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:45.719120 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c31388-892e-4685-8438-b435359db182 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:45.725328 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3442bc-7613-4dde-b698-6ba1c7009119 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:45.726097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3442bc-7613-4dde-b698-6ba1c7009119 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:45.726520 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3442bc-7613-4dde-b698-6ba1c7009119 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:45.726988 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3442bc-7613-4dde-b698-6ba1c7009119 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:45.728116 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3442bc-7613-4dde-b698-6ba1c7009119 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:45.731250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3442bc-7613-4dde-b698-6ba1c7009119 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:45.741308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 794/1572] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:56:45 2022] POST /identity/v3/users => generated 295 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:45.743009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc1ce56-7f1f-40b4-8334-7068e7d96802 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:45.757029 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc1ce56-7f1f-40b4-8334-7068e7d96802 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:45.763138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60d95ea-4147-4048-a2c3-dacf40e132cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:45.763845 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60d95ea-4147-4048-a2c3-dacf40e132cd admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:45.764196 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60d95ea-4147-4048-a2c3-dacf40e132cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:45.764552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60d95ea-4147-4048-a2c3-dacf40e132cd admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:45.765408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60d95ea-4147-4048-a2c3-dacf40e132cd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:56:45.765747 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60d95ea-4147-4048-a2c3-dacf40e132cd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:45.767818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60d95ea-4147-4048-a2c3-dacf40e132cd admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:45.771970 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 779/1573] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:56:45 2022] POST /identity/v3/projects => generated 391 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:45.773558 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f085fb76-475c-4f57-8c33-e4a6ebdfa274 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:45.773967 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f085fb76-475c-4f57-8c33-e4a6ebdfa274 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:45.774342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f085fb76-475c-4f57-8c33-e4a6ebdfa274 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:45.776029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 780/1574] 10.222.0.42 () {54 vars in 965 bytes} [Sun Sep 18 00:56:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:56:45.777390 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f085fb76-475c-4f57-8c33-e4a6ebdfa274 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:45.793832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 795/1575] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:56:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:56:45.795317 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60d95ea-4147-4048-a2c3-dacf40e132cd admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:45.816263 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60d95ea-4147-4048-a2c3-dacf40e132cd admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:45.867854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f085fb76-475c-4f57-8c33-e4a6ebdfa274 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:45.875193 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c54e85-2ecb-4d00-a3d9-f73549ba6283 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:45.876007 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c54e85-2ecb-4d00-a3d9-f73549ba6283 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:45.876369 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c54e85-2ecb-4d00-a3d9-f73549ba6283 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:45.876751 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c54e85-2ecb-4d00-a3d9-f73549ba6283 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:45.877581 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c54e85-2ecb-4d00-a3d9-f73549ba6283 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:45.882018 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c54e85-2ecb-4d00-a3d9-f73549ba6283 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:45.898229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d0ba63-ccf3-493d-a68e-1cff239107d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:45.899096 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d0ba63-ccf3-493d-a68e-1cff239107d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:45.899573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d0ba63-ccf3-493d-a68e-1cff239107d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:45.899986 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d0ba63-ccf3-493d-a68e-1cff239107d7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:45.900975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d0ba63-ccf3-493d-a68e-1cff239107d7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:45.903532 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d0ba63-ccf3-493d-a68e-1cff239107d7 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:45.917477 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 781/1576] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:56:45 2022] POST /identity/v3/users => generated 303 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:45.917477 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c54e85-2ecb-4d00-a3d9-f73549ba6283 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:45.925520 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c54e85-2ecb-4d00-a3d9-f73549ba6283 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:45.933837 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24aea05e-b60f-4f59-b08a-503e8762e7b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:45.935017 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24aea05e-b60f-4f59-b08a-503e8762e7b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:45.935511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24aea05e-b60f-4f59-b08a-503e8762e7b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:45.935982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24aea05e-b60f-4f59-b08a-503e8762e7b0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:45.937204 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24aea05e-b60f-4f59-b08a-503e8762e7b0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:56:45.937730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24aea05e-b60f-4f59-b08a-503e8762e7b0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:45.940683 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24aea05e-b60f-4f59-b08a-503e8762e7b0 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:45.945228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 796/1577] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:56:45 2022] POST /identity/v3/projects => generated 373 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:45.946769 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d0ba63-ccf3-493d-a68e-1cff239107d7 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:45.963751 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 782/1578] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:56:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:56:45.965045 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24aea05e-b60f-4f59-b08a-503e8762e7b0 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:45.980934 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24aea05e-b60f-4f59-b08a-503e8762e7b0 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:46.036642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d0ba63-ccf3-493d-a68e-1cff239107d7 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:46.042194 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca2b6fe-80c9-4e8b-89ea-63087d8202cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:46.043000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca2b6fe-80c9-4e8b-89ea-63087d8202cf admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:46.043291 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca2b6fe-80c9-4e8b-89ea-63087d8202cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:46.043553 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca2b6fe-80c9-4e8b-89ea-63087d8202cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:46.044212 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca2b6fe-80c9-4e8b-89ea-63087d8202cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:46.045867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca2b6fe-80c9-4e8b-89ea-63087d8202cf admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:46.062956 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 797/1579] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:45 2022] GET /identity/v3/roles => generated 1420 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:46.062956 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca2b6fe-80c9-4e8b-89ea-63087d8202cf admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:46.064814 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227f87a6-719c-41b3-bcee-dd6702ba67f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:46.065459 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227f87a6-719c-41b3-bcee-dd6702ba67f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:46.065754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227f87a6-719c-41b3-bcee-dd6702ba67f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:46.066100 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227f87a6-719c-41b3-bcee-dd6702ba67f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:46.066723 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227f87a6-719c-41b3-bcee-dd6702ba67f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:46.068925 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227f87a6-719c-41b3-bcee-dd6702ba67f6 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:46.072903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca2b6fe-80c9-4e8b-89ea-63087d8202cf admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:46.078812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4e2235-e4d0-449d-8cc8-d01b69488427 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:46.079618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4e2235-e4d0-449d-8cc8-d01b69488427 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:46.080043 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4e2235-e4d0-449d-8cc8-d01b69488427 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:46.080433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4e2235-e4d0-449d-8cc8-d01b69488427 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:46.080969 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 783/1580] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:45 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:46.081346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4e2235-e4d0-449d-8cc8-d01b69488427 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:46.082245 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873d08b6-900d-4965-9e3f-2cf5bf318b0c None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:46.082611 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873d08b6-900d-4965-9e3f-2cf5bf318b0c None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:46.082885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873d08b6-900d-4965-9e3f-2cf5bf318b0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:46.084831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4e2235-e4d0-449d-8cc8-d01b69488427 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:46.115951 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 798/1581] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:56:46 2022] POST /identity/v3/users => generated 296 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:46.118349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4e2235-e4d0-449d-8cc8-d01b69488427 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:46.134376 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4e2235-e4d0-449d-8cc8-d01b69488427 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:46.141903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf0f0d4-5ab2-49eb-a8cb-0c879a0f9294 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:46.142347 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-873d08b6-900d-4965-9e3f-2cf5bf318b0c None None] MFA Rules not processed for user `a59d73fd45a54651930d52300704fdb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:46.142999 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf0f0d4-5ab2-49eb-a8cb-0c879a0f9294 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:46.143386 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf0f0d4-5ab2-49eb-a8cb-0c879a0f9294 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:46.143839 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf0f0d4-5ab2-49eb-a8cb-0c879a0f9294 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:46.144422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-873d08b6-900d-4965-9e3f-2cf5bf318b0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:46.144955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf0f0d4-5ab2-49eb-a8cb-0c879a0f9294 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:46.148642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf0f0d4-5ab2-49eb-a8cb-0c879a0f9294 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:46.153997 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:46.191728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 799/1582] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:56:46 2022] POST /identity/v3/projects => generated 387 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:46.191728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf0f0d4-5ab2-49eb-a8cb-0c879a0f9294 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:46.204156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:46.246986 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 784/1583] 10.222.0.42 () {58 vars in 1062 bytes} [Sun Sep 18 00:56:46 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:46.248071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873d08b6-900d-4965-9e3f-2cf5bf318b0c None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:46.264693 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873d08b6-900d-4965-9e3f-2cf5bf318b0c None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:46.283092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf0f0d4-5ab2-49eb-a8cb-0c879a0f9294 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:46.289182 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e3212a-4bd8-401a-a8e7-d18efe9e146c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:46.289920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e3212a-4bd8-401a-a8e7-d18efe9e146c admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:46.290347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e3212a-4bd8-401a-a8e7-d18efe9e146c admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:46.290786 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e3212a-4bd8-401a-a8e7-d18efe9e146c admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:46.291678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e3212a-4bd8-401a-a8e7-d18efe9e146c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:46.293804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e3212a-4bd8-401a-a8e7-d18efe9e146c admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:46.310747 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 800/1584] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:46 2022] GET /identity/v3/roles => generated 1420 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:46.310747 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e3212a-4bd8-401a-a8e7-d18efe9e146c admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:46.320912 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e3212a-4bd8-401a-a8e7-d18efe9e146c admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:46.327723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d24529-5ee4-449b-82f7-59780644fad5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:46.328557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d24529-5ee4-449b-82f7-59780644fad5 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:46.329050 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d24529-5ee4-449b-82f7-59780644fad5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:46.329443 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d24529-5ee4-449b-82f7-59780644fad5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:46.330324 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d24529-5ee4-449b-82f7-59780644fad5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:46.332936 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d24529-5ee4-449b-82f7-59780644fad5 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:46.346503 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8418891a-221b-4194-8162-96ecf5c7da52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:46.348163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8418891a-221b-4194-8162-96ecf5c7da52 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:46.348670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8418891a-221b-4194-8162-96ecf5c7da52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:46.349132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8418891a-221b-4194-8162-96ecf5c7da52 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:46.350867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8418891a-221b-4194-8162-96ecf5c7da52 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:46.354755 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8418891a-221b-4194-8162-96ecf5c7da52 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:46.367740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 801/1585] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:56:46 2022] POST /identity/v3/projects => generated 385 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:46.367740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d24529-5ee4-449b-82f7-59780644fad5 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:46.393182 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 785/1586] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:56:46 2022] POST /identity/v3/users => generated 287 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:46.393182 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8418891a-221b-4194-8162-96ecf5c7da52 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:46.402273 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8418891a-221b-4194-8162-96ecf5c7da52 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:46.478496 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d24529-5ee4-449b-82f7-59780644fad5 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:46.485456 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0605167-79be-4d39-b1cf-dceac87dd562 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:46.485738 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da98705-21e5-4672-8112-83fd8b4be066 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:46.486236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0605167-79be-4d39-b1cf-dceac87dd562 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:46.486375 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da98705-21e5-4672-8112-83fd8b4be066 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:46.486667 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0605167-79be-4d39-b1cf-dceac87dd562 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:46.486880 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da98705-21e5-4672-8112-83fd8b4be066 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:46.487049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0605167-79be-4d39-b1cf-dceac87dd562 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:46.487308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da98705-21e5-4672-8112-83fd8b4be066 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:46.487943 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0605167-79be-4d39-b1cf-dceac87dd562 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:46.488350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da98705-21e5-4672-8112-83fd8b4be066 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:46.489924 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0605167-79be-4d39-b1cf-dceac87dd562 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:46.491703 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da98705-21e5-4672-8112-83fd8b4be066 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:46.509000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 786/1587] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:46 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:46.509000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0605167-79be-4d39-b1cf-dceac87dd562 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:46.515244 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 802/1588] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:46 2022] GET /identity/v3/roles => generated 1420 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:46.515244 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da98705-21e5-4672-8112-83fd8b4be066 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:46.519490 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0605167-79be-4d39-b1cf-dceac87dd562 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:46.526683 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31646a7d-6deb-4019-ba4c-59a60671723e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:46.527380 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31646a7d-6deb-4019-ba4c-59a60671723e admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:46.527666 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31646a7d-6deb-4019-ba4c-59a60671723e admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:46.527948 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31646a7d-6deb-4019-ba4c-59a60671723e admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:46.528653 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31646a7d-6deb-4019-ba4c-59a60671723e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:46.529342 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da98705-21e5-4672-8112-83fd8b4be066 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:46.531146 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31646a7d-6deb-4019-ba4c-59a60671723e admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:46.536615 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7735112f-66fd-4ca4-87a6-28394fa5f64f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:46.538222 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7735112f-66fd-4ca4-87a6-28394fa5f64f admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:46.538803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7735112f-66fd-4ca4-87a6-28394fa5f64f admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:46.539308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7735112f-66fd-4ca4-87a6-28394fa5f64f admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:46.540434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7735112f-66fd-4ca4-87a6-28394fa5f64f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:46.544246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7735112f-66fd-4ca4-87a6-28394fa5f64f admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:46.544644 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 787/1589] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:46 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:46.545248 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31646a7d-6deb-4019-ba4c-59a60671723e admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:46.570329 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:46.580293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 803/1590] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:56:46 2022] POST /identity/v3/users => generated 294 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:46.581742 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7735112f-66fd-4ca4-87a6-28394fa5f64f admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:46.605999 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7735112f-66fd-4ca4-87a6-28394fa5f64f admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:46.613961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aae702b-0031-418c-9a71-2f28f2dbda4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:46.614659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aae702b-0031-418c-9a71-2f28f2dbda4d admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:46.615822 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aae702b-0031-418c-9a71-2f28f2dbda4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:46.616103 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aae702b-0031-418c-9a71-2f28f2dbda4d admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:46.616776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aae702b-0031-418c-9a71-2f28f2dbda4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:46.618718 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aae702b-0031-418c-9a71-2f28f2dbda4d admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:46.636696 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 804/1591] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:46 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:46.638246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aae702b-0031-418c-9a71-2f28f2dbda4d admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:46.657664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aae702b-0031-418c-9a71-2f28f2dbda4d admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:46.657962 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31646a7d-6deb-4019-ba4c-59a60671723e admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:46.665075 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac21923-9c29-4684-a5c8-f8d53ce9793d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:46.666399 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438a85dd-44e8-4e2c-b23c-86e28abe0338 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:46.666727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac21923-9c29-4684-a5c8-f8d53ce9793d admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:46.667116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac21923-9c29-4684-a5c8-f8d53ce9793d admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:46.667417 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438a85dd-44e8-4e2c-b23c-86e28abe0338 service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:46.667566 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac21923-9c29-4684-a5c8-f8d53ce9793d admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:46.667882 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438a85dd-44e8-4e2c-b23c-86e28abe0338 service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:46.668302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438a85dd-44e8-4e2c-b23c-86e28abe0338 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:46.668660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac21923-9c29-4684-a5c8-f8d53ce9793d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:46.672279 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac21923-9c29-4684-a5c8-f8d53ce9793d admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:46.703051 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 805/1592] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:56:46 2022] POST /identity/v3/users => generated 293 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:46.704556 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac21923-9c29-4684-a5c8-f8d53ce9793d admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:46.717696 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac21923-9c29-4684-a5c8-f8d53ce9793d admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:46.725510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07b97df-ba76-4f62-b1b0-33f6696e40ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:46.725510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07b97df-ba76-4f62-b1b0-33f6696e40ef admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:46.726044 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07b97df-ba76-4f62-b1b0-33f6696e40ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:46.726488 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07b97df-ba76-4f62-b1b0-33f6696e40ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:46.727394 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07b97df-ba76-4f62-b1b0-33f6696e40ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:46.729267 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07b97df-ba76-4f62-b1b0-33f6696e40ef admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:46.744358 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 806/1593] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:46 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:46.745347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07b97df-ba76-4f62-b1b0-33f6696e40ef admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:46.757444 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438a85dd-44e8-4e2c-b23c-86e28abe0338 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:46.757827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07b97df-ba76-4f62-b1b0-33f6696e40ef admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:46.759531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438a85dd-44e8-4e2c-b23c-86e28abe0338 service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:46.764084 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff07a607-ba6f-4927-871e-17a9af61ff13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:46.764890 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff07a607-ba6f-4927-871e-17a9af61ff13 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:46.765380 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff07a607-ba6f-4927-871e-17a9af61ff13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:46.765878 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff07a607-ba6f-4927-871e-17a9af61ff13 admin admin] PATH_INFO: `/v3/projects/505bd70415a04f53a64645eb7959a5fb/users/294ae98a8492468893beb3b8c2870d5e/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:46.772425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 788/1594] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:56:46 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 228 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:56:46.773280 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438a85dd-44e8-4e2c-b23c-86e28abe0338 service nova] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:46.784425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438a85dd-44e8-4e2c-b23c-86e28abe0338 service nova] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:46.784958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff07a607-ba6f-4927-871e-17a9af61ff13 admin admin] RBAC: Authorizing `identity:create_grant(project_id=505bd70415a04f53a64645eb7959a5fb, user_id=294ae98a8492468893beb3b8c2870d5e, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:46.788073 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff07a607-ba6f-4927-871e-17a9af61ff13 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:46.790668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a347b76f-ac2d-4a69-af30-f4fc19c9214f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:46.791376 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a347b76f-ac2d-4a69-af30-f4fc19c9214f admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:46.791683 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a347b76f-ac2d-4a69-af30-f4fc19c9214f admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:46.792042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a347b76f-ac2d-4a69-af30-f4fc19c9214f admin admin] PATH_INFO: `/v3/projects/b94600e628984a14a6abc8764ea5be31/users/468d75d910874fa8bb0f4dacb589f64c/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:46.804632 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:46.813028 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a347b76f-ac2d-4a69-af30-f4fc19c9214f admin admin] RBAC: Authorizing `identity:create_grant(project_id=b94600e628984a14a6abc8764ea5be31, user_id=468d75d910874fa8bb0f4dacb589f64c, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:46.814733 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 807/1595] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:56:46 2022] PUT /identity/v3/projects/505bd70415a04f53a64645eb7959a5fb/users/294ae98a8492468893beb3b8c2870d5e/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:56:46.816245 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a347b76f-ac2d-4a69-af30-f4fc19c9214f admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:46.816390 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff07a607-ba6f-4927-871e-17a9af61ff13 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:46.832767 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 789/1596] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:56:46 2022] PUT /identity/v3/projects/b94600e628984a14a6abc8764ea5be31/users/468d75d910874fa8bb0f4dacb589f64c/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:56:46.834228 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a347b76f-ac2d-4a69-af30-f4fc19c9214f admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:46.902792 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff07a607-ba6f-4927-871e-17a9af61ff13 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:46.908903 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a347b76f-ac2d-4a69-af30-f4fc19c9214f admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:46.909148 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e8120a-e6ca-4148-b526-5ac97b912125 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:46.909760 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e8120a-e6ca-4148-b526-5ac97b912125 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:46.909981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e8120a-e6ca-4148-b526-5ac97b912125 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:46.910275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e8120a-e6ca-4148-b526-5ac97b912125 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:46.910982 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e8120a-e6ca-4148-b526-5ac97b912125 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:46.913320 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e8120a-e6ca-4148-b526-5ac97b912125 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:46.915279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70902fa7-98ca-47b9-8265-32062ac2e228 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:46.916397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70902fa7-98ca-47b9-8265-32062ac2e228 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:46.916684 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70902fa7-98ca-47b9-8265-32062ac2e228 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:46.916981 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70902fa7-98ca-47b9-8265-32062ac2e228 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:46.917640 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70902fa7-98ca-47b9-8265-32062ac2e228 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:46.920204 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70902fa7-98ca-47b9-8265-32062ac2e228 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:46.925431 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 808/1597] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:46 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:46.926129 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e8120a-e6ca-4148-b526-5ac97b912125 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:46.935581 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 790/1598] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:46 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:46.936473 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70902fa7-98ca-47b9-8265-32062ac2e228 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:46.936763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e8120a-e6ca-4148-b526-5ac97b912125 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:46.942102 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569f2f49-9564-4b03-8e9d-fc00d345ae33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:46.942908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569f2f49-9564-4b03-8e9d-fc00d345ae33 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:46.943871 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569f2f49-9564-4b03-8e9d-fc00d345ae33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:46.944148 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569f2f49-9564-4b03-8e9d-fc00d345ae33 admin admin] PATH_INFO: `/v3/projects/db898075a1c44732b8c837738c60e0a0/users/849ab641c3cf473eba263da27af79af8/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:46.948746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70902fa7-98ca-47b9-8265-32062ac2e228 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:46.954343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6af959-e8c7-4a30-a861-a83f5988a2a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:46.955171 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6af959-e8c7-4a30-a861-a83f5988a2a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:46.955559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6af959-e8c7-4a30-a861-a83f5988a2a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:46.955942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6af959-e8c7-4a30-a861-a83f5988a2a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:46.956718 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6af959-e8c7-4a30-a861-a83f5988a2a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:46.958482 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6af959-e8c7-4a30-a861-a83f5988a2a2 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:46.962860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569f2f49-9564-4b03-8e9d-fc00d345ae33 admin admin] RBAC: Authorizing `identity:create_grant(project_id=db898075a1c44732b8c837738c60e0a0, user_id=849ab641c3cf473eba263da27af79af8, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:46.966205 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569f2f49-9564-4b03-8e9d-fc00d345ae33 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:46.971828 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 791/1599] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:46 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:46.972584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6af959-e8c7-4a30-a861-a83f5988a2a2 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:46.983489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6af959-e8c7-4a30-a861-a83f5988a2a2 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:46.985716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 809/1600] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:56:46 2022] PUT /identity/v3/projects/db898075a1c44732b8c837738c60e0a0/users/849ab641c3cf473eba263da27af79af8/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:56:46.987903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1ce243-38ad-4d43-a2cd-c6b4191642d9 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:46.988311 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1ce243-38ad-4d43-a2cd-c6b4191642d9 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:46.988743 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1ce243-38ad-4d43-a2cd-c6b4191642d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:47.008754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:47.015708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-ca1ce243-38ad-4d43-a2cd-c6b4191642d9 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:47.016913 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-ca1ce243-38ad-4d43-a2cd-c6b4191642d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:47.096967 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2082d79-1c39-4bec-9f80-9e2f566ae107 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:47.097987 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2082d79-1c39-4bec-9f80-9e2f566ae107 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:47.098493 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2082d79-1c39-4bec-9f80-9e2f566ae107 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:47.099296 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2082d79-1c39-4bec-9f80-9e2f566ae107 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:47.100611 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2082d79-1c39-4bec-9f80-9e2f566ae107 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:47.103817 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2082d79-1c39-4bec-9f80-9e2f566ae107 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:47.106600 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 810/1601] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:47.108002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa271222-d5fd-429a-af7b-1004b7f15917 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:47.108568 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa271222-d5fd-429a-af7b-1004b7f15917 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:47.108998 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa271222-d5fd-429a-af7b-1004b7f15917 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:47.133265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-aa271222-d5fd-429a-af7b-1004b7f15917 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:47.134501 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-aa271222-d5fd-429a-af7b-1004b7f15917 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:47.140167 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 792/1602] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:46 2022] GET /identity/v3/roles => generated 1420 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:47.140167 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2082d79-1c39-4bec-9f80-9e2f566ae107 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:47.143032 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 811/1603] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:47.143988 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa271222-d5fd-429a-af7b-1004b7f15917 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:47.155052 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2082d79-1c39-4bec-9f80-9e2f566ae107 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:47.155941 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa271222-d5fd-429a-af7b-1004b7f15917 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:47.160444 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee50b920-633c-43e9-976f-3229379f1c87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:47.161066 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee50b920-633c-43e9-976f-3229379f1c87 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:47.161339 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee50b920-633c-43e9-976f-3229379f1c87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:47.161603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee50b920-633c-43e9-976f-3229379f1c87 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:47.162287 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee50b920-633c-43e9-976f-3229379f1c87 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:47.163309 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60109169-2603-4a8b-a223-fcfd93ca5207 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:47.163944 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee50b920-633c-43e9-976f-3229379f1c87 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:47.164384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60109169-2603-4a8b-a223-fcfd93ca5207 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:47.164858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60109169-2603-4a8b-a223-fcfd93ca5207 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:47.165336 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60109169-2603-4a8b-a223-fcfd93ca5207 admin admin] PATH_INFO: `/v3/projects/df01daf86854497fad1d1d016c269ef1/users/54d4e437036640b58fda503c4d213f5c/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:47.175652 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 812/1604] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:47 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:47.177114 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee50b920-633c-43e9-976f-3229379f1c87 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:47.192078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee50b920-633c-43e9-976f-3229379f1c87 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:47.193848 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60109169-2603-4a8b-a223-fcfd93ca5207 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df01daf86854497fad1d1d016c269ef1, user_id=54d4e437036640b58fda503c4d213f5c, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:47.199452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60109169-2603-4a8b-a223-fcfd93ca5207 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:47.200193 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cf9101-f74d-4ec5-a37e-ee21faa43664 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:47.201142 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cf9101-f74d-4ec5-a37e-ee21faa43664 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:47.201589 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cf9101-f74d-4ec5-a37e-ee21faa43664 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:47.202024 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cf9101-f74d-4ec5-a37e-ee21faa43664 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:47.203155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cf9101-f74d-4ec5-a37e-ee21faa43664 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:47.205626 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cf9101-f74d-4ec5-a37e-ee21faa43664 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:47.219939 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 793/1605] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:56:47 2022] PUT /identity/v3/projects/df01daf86854497fad1d1d016c269ef1/users/54d4e437036640b58fda503c4d213f5c/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:56:47.225847 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 813/1606] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:47 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:47.226153 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf92b02-f2f1-4229-bc8b-7fee63a09d49 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:47.226153 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf92b02-f2f1-4229-bc8b-7fee63a09d49 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:47.226153 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf92b02-f2f1-4229-bc8b-7fee63a09d49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:47.226654 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052d538e-a789-465c-9715-b2ff011be043 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:47.226654 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052d538e-a789-465c-9715-b2ff011be043 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:47.226654 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052d538e-a789-465c-9715-b2ff011be043 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:47.228117 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 814/1607] 10.222.0.42 () {54 vars in 912 bytes} [Sun Sep 18 00:56:47 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:56:47.232097 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052d538e-a789-465c-9715-b2ff011be043 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:47.263820 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-ebf92b02-f2f1-4229-bc8b-7fee63a09d49 None None] MFA Rules not processed for user `849ab641c3cf473eba263da27af79af8`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:47.265268 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ebf92b02-f2f1-4229-bc8b-7fee63a09d49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:47.298418 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:47.321775 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052d538e-a789-465c-9715-b2ff011be043 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:47.327754 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ef42ba-2c6e-4604-acd6-eebb606339c9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:47.328687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ef42ba-2c6e-4604-acd6-eebb606339c9 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:47.329104 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ef42ba-2c6e-4604-acd6-eebb606339c9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:47.329523 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ef42ba-2c6e-4604-acd6-eebb606339c9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:47.357869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 794/1608] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:47 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:47.359423 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf92b02-f2f1-4229-bc8b-7fee63a09d49 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:47.430592 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ef42ba-2c6e-4604-acd6-eebb606339c9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:47.432085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ef42ba-2c6e-4604-acd6-eebb606339c9 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:47.443420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 815/1609] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:56:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:56:47.444294 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ef42ba-2c6e-4604-acd6-eebb606339c9 service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:47.446665 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf92b02-f2f1-4229-bc8b-7fee63a09d49 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:47.452358 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e0376f-6f1e-4761-9363-cc3ad005b794 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:47.453241 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e0376f-6f1e-4761-9363-cc3ad005b794 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:47.453624 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e0376f-6f1e-4761-9363-cc3ad005b794 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:47.454021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ef42ba-2c6e-4604-acd6-eebb606339c9 service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:47.454230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e0376f-6f1e-4761-9363-cc3ad005b794 admin admin] PATH_INFO: `/v3/projects/5e3c031bf2e041d5b3257c389ad48cdd/users/501378b46fb346fda42399e5d2af4ddc/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:47.458626 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2099e635-9b75-403c-9e3e-9c7a2e5f94ad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:47.460310 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2099e635-9b75-403c-9e3e-9c7a2e5f94ad service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:47.460591 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2099e635-9b75-403c-9e3e-9c7a2e5f94ad service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:47.460869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2099e635-9b75-403c-9e3e-9c7a2e5f94ad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:47.471865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2099e635-9b75-403c-9e3e-9c7a2e5f94ad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:47.473515 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2099e635-9b75-403c-9e3e-9c7a2e5f94ad service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:47.475853 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e0376f-6f1e-4761-9363-cc3ad005b794 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5e3c031bf2e041d5b3257c389ad48cdd, user_id=501378b46fb346fda42399e5d2af4ddc, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:47.479661 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e0376f-6f1e-4761-9363-cc3ad005b794 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:47.486551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 816/1610] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:56:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:56:47.487903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2099e635-9b75-403c-9e3e-9c7a2e5f94ad service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:47.500511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:47.501474 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2099e635-9b75-403c-9e3e-9c7a2e5f94ad service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:47.518273 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 795/1611] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:56:47 2022] PUT /identity/v3/projects/5e3c031bf2e041d5b3257c389ad48cdd/users/501378b46fb346fda42399e5d2af4ddc/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:56:47.521133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e13fd3-16cf-40bf-a234-4a2d57d0f236 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:47.522079 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e13fd3-16cf-40bf-a234-4a2d57d0f236 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:47.522951 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e13fd3-16cf-40bf-a234-4a2d57d0f236 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:47.550876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-59e13fd3-16cf-40bf-a234-4a2d57d0f236 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:47.552232 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-59e13fd3-16cf-40bf-a234-4a2d57d0f236 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:47.646724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 796/1612] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:47.648260 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e13fd3-16cf-40bf-a234-4a2d57d0f236 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:47.665773 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e13fd3-16cf-40bf-a234-4a2d57d0f236 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:47.668399 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54ef234-deff-426d-9e5d-290b0e80e1e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:47.669548 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54ef234-deff-426d-9e5d-290b0e80e1e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:47.670170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54ef234-deff-426d-9e5d-290b0e80e1e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:47.670595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54ef234-deff-426d-9e5d-290b0e80e1e7 admin admin] PATH_INFO: `/v3/projects/5f2951e9bd2844deb85847ffd121ae3a/users/1f80c6fb434f46238c4999197703f98d/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:47.673280 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8e1736-59bb-4c5c-84a8-478bfde03626 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:47.675088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8e1736-59bb-4c5c-84a8-478bfde03626 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:47.675626 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8e1736-59bb-4c5c-84a8-478bfde03626 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:47.676085 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8e1736-59bb-4c5c-84a8-478bfde03626 admin admin] PATH_INFO: `/v3/projects/cf1375f9f28c43fda090a236635c4188/users/2f9751adf8d74ab59b6b0a5d00221405/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:47.703411 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8e1736-59bb-4c5c-84a8-478bfde03626 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cf1375f9f28c43fda090a236635c4188, user_id=2f9751adf8d74ab59b6b0a5d00221405, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:47.710908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54ef234-deff-426d-9e5d-290b0e80e1e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f2951e9bd2844deb85847ffd121ae3a, user_id=1f80c6fb434f46238c4999197703f98d, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:47.716160 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8e1736-59bb-4c5c-84a8-478bfde03626 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:47.716911 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54ef234-deff-426d-9e5d-290b0e80e1e7 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:47.745825 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 797/1614] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:56:47 2022] PUT /identity/v3/projects/cf1375f9f28c43fda090a236635c4188/users/2f9751adf8d74ab59b6b0a5d00221405/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:56:47.747723 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc833ce5-83db-443a-9f52-d4369dfed8c0 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:47.748138 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc833ce5-83db-443a-9f52-d4369dfed8c0 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:47.748516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc833ce5-83db-443a-9f52-d4369dfed8c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:47.752332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 817/1614] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:56:47 2022] PUT /identity/v3/projects/5f2951e9bd2844deb85847ffd121ae3a/users/1f80c6fb434f46238c4999197703f98d/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:56:47.753388 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54ef234-deff-426d-9e5d-290b0e80e1e7 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:47.773137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-bc833ce5-83db-443a-9f52-d4369dfed8c0 None None] MFA Rules not processed for user `a59d73fd45a54651930d52300704fdb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:47.774725 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-bc833ce5-83db-443a-9f52-d4369dfed8c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:47.875213 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54ef234-deff-426d-9e5d-290b0e80e1e7 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:47.883647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:47.994646 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 798/1615] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 18 00:56:47 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 248 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:47.996827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476560a1-90db-4cf1-b9cd-8c145607282f None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:47.997115 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476560a1-90db-4cf1-b9cd-8c145607282f None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:47.997392 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476560a1-90db-4cf1-b9cd-8c145607282f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:48.012885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927bd1b7-4a31-422d-b527-6af73c003c75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:48.014154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927bd1b7-4a31-422d-b527-6af73c003c75 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:48.015023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927bd1b7-4a31-422d-b527-6af73c003c75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:48.015749 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927bd1b7-4a31-422d-b527-6af73c003c75 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:48.017202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927bd1b7-4a31-422d-b527-6af73c003c75 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:48.022781 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927bd1b7-4a31-422d-b527-6af73c003c75 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:48.052172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-476560a1-90db-4cf1-b9cd-8c145607282f None None] MFA Rules not processed for user `501378b46fb346fda42399e5d2af4ddc`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:48.054760 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-476560a1-90db-4cf1-b9cd-8c145607282f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:48.085754 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 818/1616] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:56:47 2022] POST /identity/v3/projects => generated 405 bytes in 333 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:48.091951 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927bd1b7-4a31-422d-b527-6af73c003c75 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:48.184435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:48.209325 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 799/1617] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:47 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:48.210929 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd54550-7bd6-47eb-bc48-079c046206be None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:48.211220 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd54550-7bd6-47eb-bc48-079c046206be None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:48.211497 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd54550-7bd6-47eb-bc48-079c046206be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:48.234362 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-acd54550-7bd6-47eb-bc48-079c046206be None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:48.235976 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-acd54550-7bd6-47eb-bc48-079c046206be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:48.272970 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927bd1b7-4a31-422d-b527-6af73c003c75 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:48.286886 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1483c1c3-47e5-4c5e-9e99-64d08f52079a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:48.287696 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1483c1c3-47e5-4c5e-9e99-64d08f52079a service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:48.287985 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1483c1c3-47e5-4c5e-9e99-64d08f52079a service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:48.288257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1483c1c3-47e5-4c5e-9e99-64d08f52079a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:48.391641 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 800/1618] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:48.391641 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fe3828-0d3b-4fb7-a101-f640205a2458 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:48.391641 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fe3828-0d3b-4fb7-a101-f640205a2458 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:48.391641 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fe3828-0d3b-4fb7-a101-f640205a2458 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:48.464246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1483c1c3-47e5-4c5e-9e99-64d08f52079a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:48.466384 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1483c1c3-47e5-4c5e-9e99-64d08f52079a service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:48.484486 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-75fe3828-0d3b-4fb7-a101-f640205a2458 None None] MFA Rules not processed for user `1f80c6fb434f46238c4999197703f98d`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:48.487060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-75fe3828-0d3b-4fb7-a101-f640205a2458 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:48.488654 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 819/1619] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:56:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:56:48.491481 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1483c1c3-47e5-4c5e-9e99-64d08f52079a service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:48.591969 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:48.614011 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1483c1c3-47e5-4c5e-9e99-64d08f52079a service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:48.625780 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b8f4cd-6fba-4842-aab6-6f879c020ad2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:48.626763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b8f4cd-6fba-4842-aab6-6f879c020ad2 service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:48.627433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b8f4cd-6fba-4842-aab6-6f879c020ad2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:48.627879 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b8f4cd-6fba-4842-aab6-6f879c020ad2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:48.757590 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 801/1620] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:48 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 364 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:48.757590 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fe3828-0d3b-4fb7-a101-f640205a2458 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:48.777227 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fe3828-0d3b-4fb7-a101-f640205a2458 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:48.786685 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933c0c75-d231-4a2e-b8f6-af18fa0280c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:48.787941 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933c0c75-d231-4a2e-b8f6-af18fa0280c8 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:48.788106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933c0c75-d231-4a2e-b8f6-af18fa0280c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:48.788475 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933c0c75-d231-4a2e-b8f6-af18fa0280c8 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:48.789734 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933c0c75-d231-4a2e-b8f6-af18fa0280c8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:48.793914 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933c0c75-d231-4a2e-b8f6-af18fa0280c8 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:48.849899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 802/1621] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:56:48 2022] POST /identity/v3/users => generated 309 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:48.850835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933c0c75-d231-4a2e-b8f6-af18fa0280c8 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:48.866430 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933c0c75-d231-4a2e-b8f6-af18fa0280c8 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:48.872631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc545b42-61bc-4063-a73b-9b96f52f342c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:48.873273 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc545b42-61bc-4063-a73b-9b96f52f342c admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:48.873853 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc545b42-61bc-4063-a73b-9b96f52f342c admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:48.874139 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc545b42-61bc-4063-a73b-9b96f52f342c admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:48.874837 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc545b42-61bc-4063-a73b-9b96f52f342c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:48.877115 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc545b42-61bc-4063-a73b-9b96f52f342c admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:48.900503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b8f4cd-6fba-4842-aab6-6f879c020ad2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:48.902941 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b8f4cd-6fba-4842-aab6-6f879c020ad2 service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:48.923027 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 803/1622] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:56:48 2022] POST /identity/v3/projects => generated 373 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:48.924258 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc545b42-61bc-4063-a73b-9b96f52f342c admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:48.942308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 820/1623] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:56:48 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 452 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:56:48.944599 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b8f4cd-6fba-4842-aab6-6f879c020ad2 service cinder] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:48.979755 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b8f4cd-6fba-4842-aab6-6f879c020ad2 service cinder] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:49.039280 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:49.050910 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc545b42-61bc-4063-a73b-9b96f52f342c admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:49.057793 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e3813e-570e-4478-ac4f-ea93d7ea5187 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:49.060086 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e3813e-570e-4478-ac4f-ea93d7ea5187 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:49.060891 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e3813e-570e-4478-ac4f-ea93d7ea5187 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:49.061443 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e3813e-570e-4478-ac4f-ea93d7ea5187 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:49.160708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d519596-58e6-425d-bdbc-f5c18f880e50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:49.161307 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d519596-58e6-425d-bdbc-f5c18f880e50 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:49.161630 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d519596-58e6-425d-bdbc-f5c18f880e50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:49.161912 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d519596-58e6-425d-bdbc-f5c18f880e50 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:49.162647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d519596-58e6-425d-bdbc-f5c18f880e50 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:49.164653 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d519596-58e6-425d-bdbc-f5c18f880e50 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:49.216190 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 821/1624] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:56:48 2022] POST /identity/v3/projects => generated 387 bytes in 273 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:49.217303 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d519596-58e6-425d-bdbc-f5c18f880e50 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:49.292061 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e3813e-570e-4478-ac4f-ea93d7ea5187 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:49.295450 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e3813e-570e-4478-ac4f-ea93d7ea5187 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:49.339675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 804/1625] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:56:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:56:49.340890 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e3813e-570e-4478-ac4f-ea93d7ea5187 service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:49.344841 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d519596-58e6-425d-bdbc-f5c18f880e50 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:49.355363 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e3813e-570e-4478-ac4f-ea93d7ea5187 service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:49.362177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327deb0c-dfc4-43d3-bfb2-a5218e440fed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:49.363324 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327deb0c-dfc4-43d3-bfb2-a5218e440fed admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:49.363911 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3309f03-7d8b-4a5a-9094-6c3df23e0630 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:49.364080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327deb0c-dfc4-43d3-bfb2-a5218e440fed admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:49.364620 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3309f03-7d8b-4a5a-9094-6c3df23e0630 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:49.365306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327deb0c-dfc4-43d3-bfb2-a5218e440fed admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:49.367033 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327deb0c-dfc4-43d3-bfb2-a5218e440fed admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:49.367738 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3309f03-7d8b-4a5a-9094-6c3df23e0630 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:49.368103 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3309f03-7d8b-4a5a-9094-6c3df23e0630 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:49.369271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3309f03-7d8b-4a5a-9094-6c3df23e0630 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:49.370533 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327deb0c-dfc4-43d3-bfb2-a5218e440fed admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:49.370777 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3309f03-7d8b-4a5a-9094-6c3df23e0630 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:49.410788 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 805/1626] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:56:49 2022] POST /identity/v3/users => generated 293 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:49.410788 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327deb0c-dfc4-43d3-bfb2-a5218e440fed admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:49.415082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 822/1627] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:49 2022] GET /identity/v3/roles => generated 1420 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:49.416700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3309f03-7d8b-4a5a-9094-6c3df23e0630 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:49.424631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327deb0c-dfc4-43d3-bfb2-a5218e440fed admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:49.444661 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3309f03-7d8b-4a5a-9094-6c3df23e0630 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:49.464575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14468db-d38e-4bc4-a85b-e679f321dbf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:49.465348 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14468db-d38e-4bc4-a85b-e679f321dbf1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:49.465644 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14468db-d38e-4bc4-a85b-e679f321dbf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:49.466943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14468db-d38e-4bc4-a85b-e679f321dbf1 admin admin] PATH_INFO: `/v3/projects/bc54e3114505481ea8561262578d0ba0/users/91fbe4cfba3f42fbbb499442b6607024/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:49.472861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32194735-e4ee-4737-8428-bb43c7c614f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:49.473631 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:49.480267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32194735-e4ee-4737-8428-bb43c7c614f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:49.482720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32194735-e4ee-4737-8428-bb43c7c614f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:49.484129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32194735-e4ee-4737-8428-bb43c7c614f1 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:49.485419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32194735-e4ee-4737-8428-bb43c7c614f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:49.492578 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32194735-e4ee-4737-8428-bb43c7c614f1 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:49.506188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14468db-d38e-4bc4-a85b-e679f321dbf1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bc54e3114505481ea8561262578d0ba0, user_id=91fbe4cfba3f42fbbb499442b6607024, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:49.533939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14468db-d38e-4bc4-a85b-e679f321dbf1 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:49.541960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 806/1628] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:56:49 2022] POST /identity/v3/users => generated 294 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:49.543138 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32194735-e4ee-4737-8428-bb43c7c614f1 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:49.568894 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 823/1629] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:56:49 2022] PUT /identity/v3/projects/bc54e3114505481ea8561262578d0ba0/users/91fbe4cfba3f42fbbb499442b6607024/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:56:49.570211 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14468db-d38e-4bc4-a85b-e679f321dbf1 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:49.571797 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32194735-e4ee-4737-8428-bb43c7c614f1 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:49.709504 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14468db-d38e-4bc4-a85b-e679f321dbf1 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:49.716073 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99d05f8-e5ae-4fb4-bccd-32c5e2b69bf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:49.716891 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99d05f8-e5ae-4fb4-bccd-32c5e2b69bf0 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:49.718017 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99d05f8-e5ae-4fb4-bccd-32c5e2b69bf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:49.718445 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99d05f8-e5ae-4fb4-bccd-32c5e2b69bf0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:49.719223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c99d05f8-e5ae-4fb4-bccd-32c5e2b69bf0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:49.721899 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c99d05f8-e5ae-4fb4-bccd-32c5e2b69bf0 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:49.731903 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc49d6f-db4e-4d59-ad95-a3cb3681fb84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:49.732914 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc49d6f-db4e-4d59-ad95-a3cb3681fb84 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:49.734558 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc49d6f-db4e-4d59-ad95-a3cb3681fb84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:49.735048 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc49d6f-db4e-4d59-ad95-a3cb3681fb84 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:49.735922 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc49d6f-db4e-4d59-ad95-a3cb3681fb84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:49.736512 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 824/1630] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:49 2022] GET /identity/v3/roles => generated 1420 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:49.737621 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99d05f8-e5ae-4fb4-bccd-32c5e2b69bf0 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:49.738635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc49d6f-db4e-4d59-ad95-a3cb3681fb84 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:49.759703 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99d05f8-e5ae-4fb4-bccd-32c5e2b69bf0 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:49.765455 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fe3ae0-05cc-464c-a958-440e392269a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:49.766087 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fe3ae0-05cc-464c-a958-440e392269a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:49.766558 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fe3ae0-05cc-464c-a958-440e392269a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:49.766838 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fe3ae0-05cc-464c-a958-440e392269a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:49.767722 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1fe3ae0-05cc-464c-a958-440e392269a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:49.769209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1fe3ae0-05cc-464c-a958-440e392269a7 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:49.776571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 807/1631] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:49 2022] GET /identity/v3/roles => generated 1420 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:49.779161 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc49d6f-db4e-4d59-ad95-a3cb3681fb84 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:49.786173 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 825/1632] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:49 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:49.787251 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fe3ae0-05cc-464c-a958-440e392269a7 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:49.803439 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fe3ae0-05cc-464c-a958-440e392269a7 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:49.811177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc49d6f-db4e-4d59-ad95-a3cb3681fb84 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:49.811426 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5722b693-4efc-4570-b46e-418c4022e64d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:49.812326 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5722b693-4efc-4570-b46e-418c4022e64d admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:49.812812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5722b693-4efc-4570-b46e-418c4022e64d admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:49.813102 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5722b693-4efc-4570-b46e-418c4022e64d admin admin] PATH_INFO: `/v3/projects/4bb885f46fb24f71b8b03c593ee4225b/users/93bebdaa15b842d18e1ca569c46735e8/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:49.827521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cbfe84-7c31-4569-af5c-71c223dd2824 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:49.828956 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cbfe84-7c31-4569-af5c-71c223dd2824 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:49.829263 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cbfe84-7c31-4569-af5c-71c223dd2824 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:49.829817 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cbfe84-7c31-4569-af5c-71c223dd2824 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:49.830625 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3cbfe84-7c31-4569-af5c-71c223dd2824 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:49.835911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3cbfe84-7c31-4569-af5c-71c223dd2824 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:49.838686 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5722b693-4efc-4570-b46e-418c4022e64d admin admin] RBAC: Authorizing `identity:create_grant(project_id=4bb885f46fb24f71b8b03c593ee4225b, user_id=93bebdaa15b842d18e1ca569c46735e8, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:49.843209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5722b693-4efc-4570-b46e-418c4022e64d admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:49.885873 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 808/1633] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:49 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:49.886087 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 826/1634] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:56:49 2022] PUT /identity/v3/projects/4bb885f46fb24f71b8b03c593ee4225b/users/93bebdaa15b842d18e1ca569c46735e8/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:56:49.887001 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5722b693-4efc-4570-b46e-418c4022e64d admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:49.890024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cbfe84-7c31-4569-af5c-71c223dd2824 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:49.918998 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cbfe84-7c31-4569-af5c-71c223dd2824 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:50.075942 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5722b693-4efc-4570-b46e-418c4022e64d admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:50.076467 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e7b47b-a458-424a-b5b4-2665afa053a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:50.077189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e7b47b-a458-424a-b5b4-2665afa053a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:50.077470 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e7b47b-a458-424a-b5b4-2665afa053a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:50.077743 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e7b47b-a458-424a-b5b4-2665afa053a7 admin admin] PATH_INFO: `/v3/projects/d746b9566bbc451c935207207ca4da5c/users/0fd093470e0b49d69966e477691cb926/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:50.081723 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:50.102836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2238f379-4d63-45b3-900a-67087d767de6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:50.105472 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2238f379-4d63-45b3-900a-67087d767de6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:50.106011 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2238f379-4d63-45b3-900a-67087d767de6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:50.106510 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2238f379-4d63-45b3-900a-67087d767de6 admin admin] PATH_INFO: `/v3/projects/bc54e3114505481ea8561262578d0ba0/users/91fbe4cfba3f42fbbb499442b6607024/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:50.110127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2238f379-4d63-45b3-900a-67087d767de6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bc54e3114505481ea8561262578d0ba0, user_id=91fbe4cfba3f42fbbb499442b6607024, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:50.115462 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2238f379-4d63-45b3-900a-67087d767de6 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:50.124200 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e7b47b-a458-424a-b5b4-2665afa053a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d746b9566bbc451c935207207ca4da5c, user_id=0fd093470e0b49d69966e477691cb926, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:50.126913 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e7b47b-a458-424a-b5b4-2665afa053a7 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:50.146761 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 809/1635] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:56:49 2022] PUT /identity/v3/projects/bc54e3114505481ea8561262578d0ba0/users/91fbe4cfba3f42fbbb499442b6607024/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:56:50.152655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2238f379-4d63-45b3-900a-67087d767de6 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:50.155279 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 827/1636] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:56:49 2022] PUT /identity/v3/projects/d746b9566bbc451c935207207ca4da5c/users/0fd093470e0b49d69966e477691cb926/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:56:50.167514 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4539f32c-7c09-4c88-938f-7bb83ae9b57f None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:50.168086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4539f32c-7c09-4c88-938f-7bb83ae9b57f None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:50.168522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4539f32c-7c09-4c88-938f-7bb83ae9b57f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:50.225320 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-4539f32c-7c09-4c88-938f-7bb83ae9b57f None None] MFA Rules not processed for user `91fbe4cfba3f42fbbb499442b6607024`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:50.226965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-4539f32c-7c09-4c88-938f-7bb83ae9b57f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:50.263845 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:50.344983 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2238f379-4d63-45b3-900a-67087d767de6 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:50.384879 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 828/1637] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:50 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:56:50.385709 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7911b41a-3a8d-49d0-a1fe-e77e8194abd0 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:50.385709 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7911b41a-3a8d-49d0-a1fe-e77e8194abd0 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:50.385957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7911b41a-3a8d-49d0-a1fe-e77e8194abd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:50.529691 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-7911b41a-3a8d-49d0-a1fe-e77e8194abd0 None None] MFA Rules not processed for user `0fd093470e0b49d69966e477691cb926`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:50.539336 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-7911b41a-3a8d-49d0-a1fe-e77e8194abd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:50.548845 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83ac69a-c161-47a6-b9af-d09e67544c3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:50.554158 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83ac69a-c161-47a6-b9af-d09e67544c3f admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:50.554724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83ac69a-c161-47a6-b9af-d09e67544c3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:50.555157 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83ac69a-c161-47a6-b9af-d09e67544c3f admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:50.558110 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83ac69a-c161-47a6-b9af-d09e67544c3f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:50.561745 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83ac69a-c161-47a6-b9af-d09e67544c3f admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:50.593273 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 810/1638] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:50 2022] GET /identity/v3/roles => generated 1420 bytes in 446 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:50.594398 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83ac69a-c161-47a6-b9af-d09e67544c3f admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:50.769803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83ac69a-c161-47a6-b9af-d09e67544c3f admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:50.774286 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6d0965-646d-478c-9151-52352929747a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:50.775077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6d0965-646d-478c-9151-52352929747a service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:50.775489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6d0965-646d-478c-9151-52352929747a service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:50.775883 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6d0965-646d-478c-9151-52352929747a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:50.789934 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd6d0965-646d-478c-9151-52352929747a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:50.791689 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd6d0965-646d-478c-9151-52352929747a service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:50.804227 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 811/1639] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:56:50 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:56:50.805348 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6d0965-646d-478c-9151-52352929747a service cinder] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:50.817741 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6d0965-646d-478c-9151-52352929747a service cinder] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:50.825064 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f9d23b-86e2-4d2c-ad68-ccf7e69dab3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:50.825064 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f9d23b-86e2-4d2c-ad68-ccf7e69dab3b admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:50.825534 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f9d23b-86e2-4d2c-ad68-ccf7e69dab3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:50.825784 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f9d23b-86e2-4d2c-ad68-ccf7e69dab3b admin admin] PATH_INFO: `/v3/projects/4bb885f46fb24f71b8b03c593ee4225b/users/93bebdaa15b842d18e1ca569c46735e8/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:50.828303 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f9d23b-86e2-4d2c-ad68-ccf7e69dab3b admin admin] RBAC: Authorizing `identity:create_grant(project_id=4bb885f46fb24f71b8b03c593ee4225b, user_id=93bebdaa15b842d18e1ca569c46735e8, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:50.830053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 829/1640] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:50 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 450 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:50.832062 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f9d23b-86e2-4d2c-ad68-ccf7e69dab3b admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:50.833205 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e4e18e-96d5-47bf-86cc-6fe310a84e8d None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:50.833941 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e4e18e-96d5-47bf-86cc-6fe310a84e8d None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:50.835143 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e4e18e-96d5-47bf-86cc-6fe310a84e8d None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:50.838388 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 830/1641] 10.222.0.42 () {54 vars in 965 bytes} [Sun Sep 18 00:56:50 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:56:50.840514 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e4e18e-96d5-47bf-86cc-6fe310a84e8d None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:50.854667 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 812/1642] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:56:50 2022] PUT /identity/v3/projects/4bb885f46fb24f71b8b03c593ee4225b/users/93bebdaa15b842d18e1ca569c46735e8/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:56:50.856497 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8f1542-fde8-4f7e-bf8c-73572b97b0ab None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:50.856882 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8f1542-fde8-4f7e-bf8c-73572b97b0ab None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:50.857284 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8f1542-fde8-4f7e-bf8c-73572b97b0ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:50.884989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e4e18e-96d5-47bf-86cc-6fe310a84e8d None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:50.887332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-9a8f1542-fde8-4f7e-bf8c-73572b97b0ab None None] MFA Rules not processed for user `a59d73fd45a54651930d52300704fdb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:50.896174 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-9a8f1542-fde8-4f7e-bf8c-73572b97b0ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:50.928425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:51.052403 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9cf58b-64c5-4967-87a1-98c07e95c151 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:51.053661 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9cf58b-64c5-4967-87a1-98c07e95c151 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:51.053923 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9cf58b-64c5-4967-87a1-98c07e95c151 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:51.054302 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9cf58b-64c5-4967-87a1-98c07e95c151 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:51.060533 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c9cf58b-64c5-4967-87a1-98c07e95c151 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:51.062878 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c9cf58b-64c5-4967-87a1-98c07e95c151 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:51.136213 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 813/1643] 10.222.0.42 () {58 vars in 1062 bytes} [Sun Sep 18 00:56:50 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:51.140901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8433208c-cc30-4ec1-82c6-51eb6ee13611 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:51.141867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8433208c-cc30-4ec1-82c6-51eb6ee13611 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:51.142393 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8433208c-cc30-4ec1-82c6-51eb6ee13611 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:51.144986 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 831/1644] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:56:50 2022] POST /identity/v3/projects => generated 385 bytes in 306 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:51.147565 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e8e600-0cbe-44bc-b03c-28e6410008ce None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:51.148184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e8e600-0cbe-44bc-b03c-28e6410008ce None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:51.148608 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e8e600-0cbe-44bc-b03c-28e6410008ce None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:51.150303 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 832/1645] 10.222.0.42 () {54 vars in 912 bytes} [Sun Sep 18 00:56:51 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:56:51.157958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e8e600-0cbe-44bc-b03c-28e6410008ce None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:51.244440 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-8433208c-cc30-4ec1-82c6-51eb6ee13611 None None] MFA Rules not processed for user `93bebdaa15b842d18e1ca569c46735e8`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:51.247036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8433208c-cc30-4ec1-82c6-51eb6ee13611 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:51.264825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:51.348991 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e8e600-0cbe-44bc-b03c-28e6410008ce None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:51.361850 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1995070-09a5-46bd-b29e-4283c249c4ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:51.362794 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1995070-09a5-46bd-b29e-4283c249c4ce admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:51.363134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1995070-09a5-46bd-b29e-4283c249c4ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:51.363487 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1995070-09a5-46bd-b29e-4283c249c4ce admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:51.364264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1995070-09a5-46bd-b29e-4283c249c4ce admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:51.366964 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1995070-09a5-46bd-b29e-4283c249c4ce admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:51.403570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 814/1646] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:51 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 264 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:51.408831 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17f2097-e056-43c9-a12d-2e3fa381e4c7 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:51.409219 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17f2097-e056-43c9-a12d-2e3fa381e4c7 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:51.409583 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17f2097-e056-43c9-a12d-2e3fa381e4c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:51.427908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 833/1647] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:56:51 2022] POST /identity/v3/users => generated 299 bytes in 268 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:51.427908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1995070-09a5-46bd-b29e-4283c249c4ce admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:51.457736 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1995070-09a5-46bd-b29e-4283c249c4ce admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:51.483100 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387f9e81-b170-44e2-9bec-fe5f2764090f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:51.484716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387f9e81-b170-44e2-9bec-fe5f2764090f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:51.484716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387f9e81-b170-44e2-9bec-fe5f2764090f admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:51.485392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387f9e81-b170-44e2-9bec-fe5f2764090f admin admin] PATH_INFO: `/v3/users/849ab641c3cf473eba263da27af79af8` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:51.485865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-f17f2097-e056-43c9-a12d-2e3fa381e4c7 None None] MFA Rules not processed for user `a59d73fd45a54651930d52300704fdb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:51.487878 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387f9e81-b170-44e2-9bec-fe5f2764090f admin admin] RBAC: Authorizing `identity:delete_user(user_id=849ab641c3cf473eba263da27af79af8)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:51.488549 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-f17f2097-e056-43c9-a12d-2e3fa381e4c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:51.491751 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387f9e81-b170-44e2-9bec-fe5f2764090f admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:51.634610 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-387f9e81-b170-44e2-9bec-fe5f2764090f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '849ab641c3cf473eba263da27af79af8'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:51.649556 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-387f9e81-b170-44e2-9bec-fe5f2764090f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '849ab641c3cf473eba263da27af79af8'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:51.673734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-387f9e81-b170-44e2-9bec-fe5f2764090f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '849ab641c3cf473eba263da27af79af8'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:51.676588 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-387f9e81-b170-44e2-9bec-fe5f2764090f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '849ab641c3cf473eba263da27af79af8'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:51.696130 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-387f9e81-b170-44e2-9bec-fe5f2764090f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '849ab641c3cf473eba263da27af79af8'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:51.714729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 834/1648] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:56:51 2022] DELETE /identity/v3/users/849ab641c3cf473eba263da27af79af8 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:56:51.714729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387f9e81-b170-44e2-9bec-fe5f2764090f admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:51.716081 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-387f9e81-b170-44e2-9bec-fe5f2764090f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:51.732103 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 815/1649] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 18 00:56:51 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 328 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:51.733917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17f2097-e056-43c9-a12d-2e3fa381e4c7 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:51.737905 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-f17f2097-e056-43c9-a12d-2e3fa381e4c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:51.845612 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:51.863371 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387f9e81-b170-44e2-9bec-fe5f2764090f admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:51.873458 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a404636-0670-4083-a3d3-8e56b4c70aa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:51.874190 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a404636-0670-4083-a3d3-8e56b4c70aa8 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:51.874563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a404636-0670-4083-a3d3-8e56b4c70aa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:51.874903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a404636-0670-4083-a3d3-8e56b4c70aa8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:51.878031 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a404636-0670-4083-a3d3-8e56b4c70aa8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:51.882438 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a404636-0670-4083-a3d3-8e56b4c70aa8 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:51.906580 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 835/1650] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:51 2022] GET /identity/v3/roles => generated 1420 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:51.911805 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b02aeeb-a73a-4e42-aebe-4b888f7d1a9d None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:51.911805 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b02aeeb-a73a-4e42-aebe-4b888f7d1a9d None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:51.913588 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b02aeeb-a73a-4e42-aebe-4b888f7d1a9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:51.947554 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17f2097-e056-43c9-a12d-2e3fa381e4c7 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:51.963665 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-6b02aeeb-a73a-4e42-aebe-4b888f7d1a9d None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:51.965043 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-6b02aeeb-a73a-4e42-aebe-4b888f7d1a9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:51.977675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 836/1651] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:51.978726 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b02aeeb-a73a-4e42-aebe-4b888f7d1a9d None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:51.981559 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-6b02aeeb-a73a-4e42-aebe-4b888f7d1a9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:52.007789 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:52.092819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347d5a5e-1cf9-4f2c-9574-1ff31fc7b54b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:52.097879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347d5a5e-1cf9-4f2c-9574-1ff31fc7b54b service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:52.098360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347d5a5e-1cf9-4f2c-9574-1ff31fc7b54b service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:52.098892 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347d5a5e-1cf9-4f2c-9574-1ff31fc7b54b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:52.102854 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b02aeeb-a73a-4e42-aebe-4b888f7d1a9d None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:52.108695 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d36c80-3158-440f-8173-896351be252a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:52.109170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-347d5a5e-1cf9-4f2c-9574-1ff31fc7b54b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:52.109732 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d36c80-3158-440f-8173-896351be252a service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:52.110227 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d36c80-3158-440f-8173-896351be252a service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:52.110699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d36c80-3158-440f-8173-896351be252a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:52.217658 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347d5a5e-1cf9-4f2c-9574-1ff31fc7b54b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:52.220164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347d5a5e-1cf9-4f2c-9574-1ff31fc7b54b service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:52.248392 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 816/1652] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:56:51 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 515 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:56:52.250040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347d5a5e-1cf9-4f2c-9574-1ff31fc7b54b service cinder] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:52.269056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d36c80-3158-440f-8173-896351be252a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:52.274213 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d36c80-3158-440f-8173-896351be252a service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:52.290626 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347d5a5e-1cf9-4f2c-9574-1ff31fc7b54b service cinder] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:52.302194 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a43345-e876-44af-bc8f-1c785a108ab5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:52.303531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a43345-e876-44af-bc8f-1c785a108ab5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:52.304015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a43345-e876-44af-bc8f-1c785a108ab5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:52.304507 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a43345-e876-44af-bc8f-1c785a108ab5 admin admin] PATH_INFO: `/v3/projects/378035cecce241af8f09636c8f412eb8/users/b2801b77031544e7ad0bcafca672bfd8/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:52.306766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 837/1653] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:56:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:56:52.307672 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d36c80-3158-440f-8173-896351be252a service cinder] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:52.311116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-b1d36c80-3158-440f-8173-896351be252a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:52.364964 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a43345-e876-44af-bc8f-1c785a108ab5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=378035cecce241af8f09636c8f412eb8, user_id=b2801b77031544e7ad0bcafca672bfd8, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:52.376519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a43345-e876-44af-bc8f-1c785a108ab5 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:52.448137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 817/1654] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:56:52 2022] PUT /identity/v3/projects/378035cecce241af8f09636c8f412eb8/users/b2801b77031544e7ad0bcafca672bfd8/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:56:52.448137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a43345-e876-44af-bc8f-1c785a108ab5 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:52.555799 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d36c80-3158-440f-8173-896351be252a service cinder] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:52.717604 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a43345-e876-44af-bc8f-1c785a108ab5 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:52.729507 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb13fa2-1a5a-46a2-b662-9a6890124ba7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:52.731591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb13fa2-1a5a-46a2-b662-9a6890124ba7 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:52.732617 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb13fa2-1a5a-46a2-b662-9a6890124ba7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:52.733295 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb13fa2-1a5a-46a2-b662-9a6890124ba7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:52.734793 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb13fa2-1a5a-46a2-b662-9a6890124ba7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:52.738889 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb13fa2-1a5a-46a2-b662-9a6890124ba7 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:52.766718 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 818/1655] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:52 2022] GET /identity/v3/roles => generated 1420 bytes in 340 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:52.768767 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb13fa2-1a5a-46a2-b662-9a6890124ba7 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:52.773615 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6a4907-b94f-4e67-a12e-cd9b661c337f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:52.774836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6a4907-b94f-4e67-a12e-cd9b661c337f service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:52.775125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6a4907-b94f-4e67-a12e-cd9b661c337f service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:52.775445 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6a4907-b94f-4e67-a12e-cd9b661c337f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:52.779241 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:52.798976 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6a4907-b94f-4e67-a12e-cd9b661c337f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:52.803979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6a4907-b94f-4e67-a12e-cd9b661c337f service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:52.875086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 838/1656] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:56:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 568 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:56:52.878918 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6a4907-b94f-4e67-a12e-cd9b661c337f service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:52.913344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6a4907-b94f-4e67-a12e-cd9b661c337f service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:52.927662 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55467a3-4e6b-49b8-9958-112650ac5387 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:52.930605 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55467a3-4e6b-49b8-9958-112650ac5387 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:52.930914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55467a3-4e6b-49b8-9958-112650ac5387 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:52.931200 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55467a3-4e6b-49b8-9958-112650ac5387 admin admin] PATH_INFO: `/v3/projects/378035cecce241af8f09636c8f412eb8/users/b2801b77031544e7ad0bcafca672bfd8/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:52.935418 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55467a3-4e6b-49b8-9958-112650ac5387 admin admin] RBAC: Authorizing `identity:create_grant(project_id=378035cecce241af8f09636c8f412eb8, user_id=b2801b77031544e7ad0bcafca672bfd8, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:52.943726 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55467a3-4e6b-49b8-9958-112650ac5387 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:52.978718 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 839/1657] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:56:52 2022] PUT /identity/v3/projects/378035cecce241af8f09636c8f412eb8/users/b2801b77031544e7ad0bcafca672bfd8/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:56:52.993323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e9b941-2279-41b2-86c0-582a9a2e3ef3 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:52.993696 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e9b941-2279-41b2-86c0-582a9a2e3ef3 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:52.993984 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e9b941-2279-41b2-86c0-582a9a2e3ef3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:53.014947 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:53.064396 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb13fa2-1a5a-46a2-b662-9a6890124ba7 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:53.129191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-71e9b941-2279-41b2-86c0-582a9a2e3ef3 None None] MFA Rules not processed for user `b2801b77031544e7ad0bcafca672bfd8`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:53.140027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-71e9b941-2279-41b2-86c0-582a9a2e3ef3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:53.299268 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ea6cb0-d7cf-43c3-be58-6c1f085ec0f0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:53.300236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ea6cb0-d7cf-43c3-be58-6c1f085ec0f0 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:53.300685 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ea6cb0-d7cf-43c3-be58-6c1f085ec0f0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:53.301150 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ea6cb0-d7cf-43c3-be58-6c1f085ec0f0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:53.308125 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-28ea6cb0-d7cf-43c3-be58-6c1f085ec0f0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:53.404662 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 840/1658] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:52 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 412 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:56:53.434319 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e9b941-2279-41b2-86c0-582a9a2e3ef3 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:53.452081 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e9b941-2279-41b2-86c0-582a9a2e3ef3 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:53.468569 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eddf46f-4507-466e-b52c-ae6a2ae5984b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:53.469688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eddf46f-4507-466e-b52c-ae6a2ae5984b service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:53.470682 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eddf46f-4507-466e-b52c-ae6a2ae5984b service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:53.471645 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eddf46f-4507-466e-b52c-ae6a2ae5984b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:53.476755 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ea6cb0-d7cf-43c3-be58-6c1f085ec0f0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:53.480845 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ea6cb0-d7cf-43c3-be58-6c1f085ec0f0 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:53.498252 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 819/1659] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:56:52 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 730 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:56:53.502179 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eddf46f-4507-466e-b52c-ae6a2ae5984b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:53.504045 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eddf46f-4507-466e-b52c-ae6a2ae5984b service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:53.536029 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 841/1660] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:56:53 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:56:54.116976 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ea6cb0-d7cf-43c3-be58-6c1f085ec0f0 service cinder] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:54.122293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-28ea6cb0-d7cf-43c3-be58-6c1f085ec0f0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:54.296085 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ea6cb0-d7cf-43c3-be58-6c1f085ec0f0 service cinder] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:54.310193 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064ffe3f-07c7-4dd7-b75c-784651011699 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:54.311342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064ffe3f-07c7-4dd7-b75c-784651011699 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:54.311847 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064ffe3f-07c7-4dd7-b75c-784651011699 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:54.312356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064ffe3f-07c7-4dd7-b75c-784651011699 admin admin] PATH_INFO: `/v3/users/30aa97b3fe804a90b252b3046d3ff401` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:54.317305 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064ffe3f-07c7-4dd7-b75c-784651011699 admin admin] RBAC: Authorizing `identity:delete_user(user_id=30aa97b3fe804a90b252b3046d3ff401)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:54.319281 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: WARNING py.warnings [None req-064ffe3f-07c7-4dd7-b75c-784651011699 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 18 00:56:54.319281 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: warnings.warn(msg) Sep 18 00:56:54.319281 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: Sep 18 00:56:54.321255 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064ffe3f-07c7-4dd7-b75c-784651011699 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:54.379759 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:54.538979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-064ffe3f-07c7-4dd7-b75c-784651011699 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '30aa97b3fe804a90b252b3046d3ff401'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:54.555222 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eddf46f-4507-466e-b52c-ae6a2ae5984b service cinder] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:54.557570 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-9eddf46f-4507-466e-b52c-ae6a2ae5984b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:54.564662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-064ffe3f-07c7-4dd7-b75c-784651011699 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '30aa97b3fe804a90b252b3046d3ff401'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:54.575520 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-064ffe3f-07c7-4dd7-b75c-784651011699 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '30aa97b3fe804a90b252b3046d3ff401'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:54.576775 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-064ffe3f-07c7-4dd7-b75c-784651011699 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '30aa97b3fe804a90b252b3046d3ff401'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:54.595008 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eddf46f-4507-466e-b52c-ae6a2ae5984b service cinder] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:54.610727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e862ead-bb47-4ea9-a84e-c09aa146ce9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:54.611938 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e862ead-bb47-4ea9-a84e-c09aa146ce9d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:54.611938 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e862ead-bb47-4ea9-a84e-c09aa146ce9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:54.611938 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e862ead-bb47-4ea9-a84e-c09aa146ce9d admin admin] PATH_INFO: `/v3/projects/db898075a1c44732b8c837738c60e0a0` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:54.613450 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-064ffe3f-07c7-4dd7-b75c-784651011699 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '30aa97b3fe804a90b252b3046d3ff401'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:54.616428 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e862ead-bb47-4ea9-a84e-c09aa146ce9d admin admin] RBAC: Authorizing `identity:delete_project(project_id=db898075a1c44732b8c837738c60e0a0)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:54.617741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: WARNING py.warnings [None req-7e862ead-bb47-4ea9-a84e-c09aa146ce9d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 18 00:56:54.617741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: warnings.warn(msg) Sep 18 00:56:54.617741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: Sep 18 00:56:54.621599 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e862ead-bb47-4ea9-a84e-c09aa146ce9d admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:54.727990 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 820/1661] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:56:54 2022] DELETE /identity/v3/users/30aa97b3fe804a90b252b3046d3ff401 => generated 0 bytes in 613 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:56:54.738580 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c2824b-b014-405f-b557-1b5e5583d691 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:54.739856 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c2824b-b014-405f-b557-1b5e5583d691 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:54.740718 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c2824b-b014-405f-b557-1b5e5583d691 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:54.797576 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-48c2824b-b014-405f-b557-1b5e5583d691 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:54.812650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-48c2824b-b014-405f-b557-1b5e5583d691 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:54.837221 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7e862ead-bb47-4ea9-a84e-c09aa146ce9d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'db898075a1c44732b8c837738c60e0a0', 'request_id': 'req-7e862ead-bb47-4ea9-a84e-c09aa146ce9d'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:54.848794 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7e862ead-bb47-4ea9-a84e-c09aa146ce9d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'db898075a1c44732b8c837738c60e0a0', 'request_id': 'req-7e862ead-bb47-4ea9-a84e-c09aa146ce9d'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:54.864102 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7e862ead-bb47-4ea9-a84e-c09aa146ce9d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'db898075a1c44732b8c837738c60e0a0', 'request_id': 'req-7e862ead-bb47-4ea9-a84e-c09aa146ce9d'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:54.948911 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7e862ead-bb47-4ea9-a84e-c09aa146ce9d admin admin] The token cache is being invalidate because project db898075a1c44732b8c837738c60e0a0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:56:54.949532 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7e862ead-bb47-4ea9-a84e-c09aa146ce9d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:54.952737 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7e862ead-bb47-4ea9-a84e-c09aa146ce9d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:54.957416 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 842/1662] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:56:54 2022] DELETE /identity/v3/projects/db898075a1c44732b8c837738c60e0a0 => generated 0 bytes in 401 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:56:54.961559 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e862ead-bb47-4ea9-a84e-c09aa146ce9d admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:54.964934 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-7e862ead-bb47-4ea9-a84e-c09aa146ce9d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:55.108643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 821/1663] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:55.109569 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c2824b-b014-405f-b557-1b5e5583d691 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:55.116353 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-48c2824b-b014-405f-b557-1b5e5583d691 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:55.238162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e862ead-bb47-4ea9-a84e-c09aa146ce9d admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:55.262883 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00b269e-2312-4306-8be7-0a5943075121 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:55.264012 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00b269e-2312-4306-8be7-0a5943075121 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:55.264526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00b269e-2312-4306-8be7-0a5943075121 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:55.265001 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00b269e-2312-4306-8be7-0a5943075121 admin admin] PATH_INFO: `/v3/users/91fbe4cfba3f42fbbb499442b6607024` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:55.275157 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00b269e-2312-4306-8be7-0a5943075121 admin admin] RBAC: Authorizing `identity:delete_user(user_id=91fbe4cfba3f42fbbb499442b6607024)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:55.278904 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00b269e-2312-4306-8be7-0a5943075121 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:55.281949 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:55.344293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c2824b-b014-405f-b557-1b5e5583d691 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:55.356280 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73cee9b4-069a-4c54-ae16-dcaa67921b0d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:55.357467 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cee9b4-069a-4c54-ae16-dcaa67921b0d service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:55.357847 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cee9b4-069a-4c54-ae16-dcaa67921b0d service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:55.358326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cee9b4-069a-4c54-ae16-dcaa67921b0d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:55.368224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-73cee9b4-069a-4c54-ae16-dcaa67921b0d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:55.478512 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a00b269e-2312-4306-8be7-0a5943075121 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '91fbe4cfba3f42fbbb499442b6607024'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:55.513018 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a00b269e-2312-4306-8be7-0a5943075121 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '91fbe4cfba3f42fbbb499442b6607024'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:55.577252 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a00b269e-2312-4306-8be7-0a5943075121 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '91fbe4cfba3f42fbbb499442b6607024'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:55.581957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a00b269e-2312-4306-8be7-0a5943075121 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '91fbe4cfba3f42fbbb499442b6607024'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:55.602702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:55.613837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a00b269e-2312-4306-8be7-0a5943075121 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '91fbe4cfba3f42fbbb499442b6607024'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:55.642130 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 843/1664] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:56:54 2022] DELETE /identity/v3/users/91fbe4cfba3f42fbbb499442b6607024 => generated 0 bytes in 678 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:56:55.643585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00b269e-2312-4306-8be7-0a5943075121 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:55.651379 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73cee9b4-069a-4c54-ae16-dcaa67921b0d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:55.654975 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73cee9b4-069a-4c54-ae16-dcaa67921b0d service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:55.655298 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a00b269e-2312-4306-8be7-0a5943075121 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:55.683366 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 822/1665] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:56:55 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 574 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:56:55.810645 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00b269e-2312-4306-8be7-0a5943075121 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:55.822205 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25acaa31-db4d-4140-99db-48094d9ac4a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:55.822880 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25acaa31-db4d-4140-99db-48094d9ac4a8 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:55.823164 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25acaa31-db4d-4140-99db-48094d9ac4a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:55.823439 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25acaa31-db4d-4140-99db-48094d9ac4a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:55.828313 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-25acaa31-db4d-4140-99db-48094d9ac4a8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:56.003286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25acaa31-db4d-4140-99db-48094d9ac4a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:56.004813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25acaa31-db4d-4140-99db-48094d9ac4a8 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:56.031884 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 844/1666] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:56:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:56:57.296480 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73cee9b4-069a-4c54-ae16-dcaa67921b0d service cinder] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:57.298427 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-73cee9b4-069a-4c54-ae16-dcaa67921b0d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:57.336271 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73cee9b4-069a-4c54-ae16-dcaa67921b0d service cinder] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:57.351017 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48bfe5ff-863b-49a5-b479-327efd9b43c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:57.351705 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bfe5ff-863b-49a5-b479-327efd9b43c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:57.351990 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bfe5ff-863b-49a5-b479-327efd9b43c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:57.352266 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bfe5ff-863b-49a5-b479-327efd9b43c3 admin admin] PATH_INFO: `/v3/projects/ee923b3faeca4113af384cc85257b516` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:57.359254 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48bfe5ff-863b-49a5-b479-327efd9b43c3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ee923b3faeca4113af384cc85257b516)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:57.363158 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48bfe5ff-863b-49a5-b479-327efd9b43c3 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:57.451951 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25acaa31-db4d-4140-99db-48094d9ac4a8 service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:57.454791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-25acaa31-db4d-4140-99db-48094d9ac4a8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:57.494673 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25acaa31-db4d-4140-99db-48094d9ac4a8 service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:57.533133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-48bfe5ff-863b-49a5-b479-327efd9b43c3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ee923b3faeca4113af384cc85257b516', 'request_id': 'req-48bfe5ff-863b-49a5-b479-327efd9b43c3'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:57.550621 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-48bfe5ff-863b-49a5-b479-327efd9b43c3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ee923b3faeca4113af384cc85257b516', 'request_id': 'req-48bfe5ff-863b-49a5-b479-327efd9b43c3'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:57.570595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-48bfe5ff-863b-49a5-b479-327efd9b43c3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ee923b3faeca4113af384cc85257b516', 'request_id': 'req-48bfe5ff-863b-49a5-b479-327efd9b43c3'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:57.591527 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:57.598452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-48bfe5ff-863b-49a5-b479-327efd9b43c3 admin admin] The token cache is being invalidate because project ee923b3faeca4113af384cc85257b516 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:56:57.598779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-48bfe5ff-863b-49a5-b479-327efd9b43c3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:57.599609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-48bfe5ff-863b-49a5-b479-327efd9b43c3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:57.610943 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 823/1667] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:56:57 2022] DELETE /identity/v3/projects/ee923b3faeca4113af384cc85257b516 => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:56:57.613732 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f51211-9a0d-4076-a16e-57718642402f None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:57.614130 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f51211-9a0d-4076-a16e-57718642402f None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:57.614417 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f51211-9a0d-4076-a16e-57718642402f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:57.666137 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812175ef-d966-4de0-a9bf-33e04c772811 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:57.666833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812175ef-d966-4de0-a9bf-33e04c772811 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:57.667116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812175ef-d966-4de0-a9bf-33e04c772811 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:57.667398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812175ef-d966-4de0-a9bf-33e04c772811 admin admin] PATH_INFO: `/v3/projects/bc54e3114505481ea8561262578d0ba0` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:57.669842 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812175ef-d966-4de0-a9bf-33e04c772811 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bc54e3114505481ea8561262578d0ba0)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:57.673320 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812175ef-d966-4de0-a9bf-33e04c772811 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:57.679482 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-04f51211-9a0d-4076-a16e-57718642402f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:57.682742 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-04f51211-9a0d-4076-a16e-57718642402f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:57.709733 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 824/1668] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:57.712989 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77d827f-f1ff-4950-8a37-ac5f5990b012 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:57.713343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77d827f-f1ff-4950-8a37-ac5f5990b012 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:57.713665 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77d827f-f1ff-4950-8a37-ac5f5990b012 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:57.767172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-d77d827f-f1ff-4950-8a37-ac5f5990b012 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:57.769058 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d77d827f-f1ff-4950-8a37-ac5f5990b012 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:57.777168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-812175ef-d966-4de0-a9bf-33e04c772811 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bc54e3114505481ea8561262578d0ba0', 'request_id': 'req-812175ef-d966-4de0-a9bf-33e04c772811'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:57.789935 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-812175ef-d966-4de0-a9bf-33e04c772811 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bc54e3114505481ea8561262578d0ba0', 'request_id': 'req-812175ef-d966-4de0-a9bf-33e04c772811'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:57.807134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-812175ef-d966-4de0-a9bf-33e04c772811 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bc54e3114505481ea8561262578d0ba0', 'request_id': 'req-812175ef-d966-4de0-a9bf-33e04c772811'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:57.828840 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-812175ef-d966-4de0-a9bf-33e04c772811 admin admin] The token cache is being invalidate because project bc54e3114505481ea8561262578d0ba0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:56:57.829163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-812175ef-d966-4de0-a9bf-33e04c772811 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:57.830679 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-812175ef-d966-4de0-a9bf-33e04c772811 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:56:57.838152 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 845/1669] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:56:57 2022] DELETE /identity/v3/projects/bc54e3114505481ea8561262578d0ba0 => generated 0 bytes in 384 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:56:57.841370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74dc715-55e9-48eb-98a3-a8533d714f6f None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:57.841671 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74dc715-55e9-48eb-98a3-a8533d714f6f None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:57.843152 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74dc715-55e9-48eb-98a3-a8533d714f6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:57.891863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-e74dc715-55e9-48eb-98a3-a8533d714f6f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:57.893223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-e74dc715-55e9-48eb-98a3-a8533d714f6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:57.988056 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 825/1670] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:57.990675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77d827f-f1ff-4950-8a37-ac5f5990b012 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:58.024892 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77d827f-f1ff-4950-8a37-ac5f5990b012 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:58.038422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad82c087-3ebe-433e-a613-588fe0053baa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:58.039720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad82c087-3ebe-433e-a613-588fe0053baa admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:58.041874 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad82c087-3ebe-433e-a613-588fe0053baa admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:58.051087 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad82c087-3ebe-433e-a613-588fe0053baa admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:58.053072 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad82c087-3ebe-433e-a613-588fe0053baa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:56:58.053072 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad82c087-3ebe-433e-a613-588fe0053baa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:58.055550 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad82c087-3ebe-433e-a613-588fe0053baa admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:58.060307 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 846/1671] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:58.069636 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf71089-0b54-4d16-90b4-84e245e277b1 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:58.069956 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf71089-0b54-4d16-90b4-84e245e277b1 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:58.070562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf71089-0b54-4d16-90b4-84e245e277b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:58.098276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 826/1672] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:56:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:56:58.100109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad82c087-3ebe-433e-a613-588fe0053baa admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:58.114110 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-bcf71089-0b54-4d16-90b4-84e245e277b1 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:58.115499 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-bcf71089-0b54-4d16-90b4-84e245e277b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:58.122379 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad82c087-3ebe-433e-a613-588fe0053baa admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:58.133660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfad8807-cd93-4f11-9b59-a56f668e2cb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:58.134303 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfad8807-cd93-4f11-9b59-a56f668e2cb1 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:58.134613 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfad8807-cd93-4f11-9b59-a56f668e2cb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:58.135365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfad8807-cd93-4f11-9b59-a56f668e2cb1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:58.137341 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 847/1673] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:58.137472 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfad8807-cd93-4f11-9b59-a56f668e2cb1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:58.140310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfad8807-cd93-4f11-9b59-a56f668e2cb1 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:58.141918 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf71089-0b54-4d16-90b4-84e245e277b1 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:58.164860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf71089-0b54-4d16-90b4-84e245e277b1 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:58.179545 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cbe1d5-476c-4b6c-9827-d9b68f0edfdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:58.181101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cbe1d5-476c-4b6c-9827-d9b68f0edfdf admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:58.182498 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cbe1d5-476c-4b6c-9827-d9b68f0edfdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:58.182800 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cbe1d5-476c-4b6c-9827-d9b68f0edfdf admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:58.185891 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31cbe1d5-476c-4b6c-9827-d9b68f0edfdf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:56:58.186360 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31cbe1d5-476c-4b6c-9827-d9b68f0edfdf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:58.193633 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31cbe1d5-476c-4b6c-9827-d9b68f0edfdf admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:58.232247 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 827/1674] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:56:58 2022] POST /identity/v3/projects => generated 383 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:58.238773 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfad8807-cd93-4f11-9b59-a56f668e2cb1 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:58.257943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 848/1675] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:56:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:56:58.262917 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cbe1d5-476c-4b6c-9827-d9b68f0edfdf admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:58.293490 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cbe1d5-476c-4b6c-9827-d9b68f0edfdf admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:58.410366 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:58.509427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:58.566940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfad8807-cd93-4f11-9b59-a56f668e2cb1 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:58.583569 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55babd57-1bdd-4edc-902a-7cba9325da49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:58.584928 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55babd57-1bdd-4edc-902a-7cba9325da49 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:58.585214 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55babd57-1bdd-4edc-902a-7cba9325da49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:58.588553 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55babd57-1bdd-4edc-902a-7cba9325da49 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:58.590342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55babd57-1bdd-4edc-902a-7cba9325da49 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:58.599139 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55babd57-1bdd-4edc-902a-7cba9325da49 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:58.608729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8702b94b-4598-4688-86ca-d5cad9b691ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:58.616414 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8702b94b-4598-4688-86ca-d5cad9b691ca admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:58.616857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8702b94b-4598-4688-86ca-d5cad9b691ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:58.617242 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8702b94b-4598-4688-86ca-d5cad9b691ca admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:58.618025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8702b94b-4598-4688-86ca-d5cad9b691ca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:58.624681 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8702b94b-4598-4688-86ca-d5cad9b691ca admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:58.679029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 828/1676] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:56:58 2022] POST /identity/v3/users => generated 292 bytes in 443 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:58.683563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55babd57-1bdd-4edc-902a-7cba9325da49 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:58.718046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55babd57-1bdd-4edc-902a-7cba9325da49 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:58.731283 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 849/1677] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:56:58 2022] POST /identity/v3/projects => generated 357 bytes in 464 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:58.733066 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8702b94b-4598-4688-86ca-d5cad9b691ca admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:58.880332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed88c035-70ea-4b36-8401-22930e6c2ea5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:58.881454 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed88c035-70ea-4b36-8401-22930e6c2ea5 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:58.881454 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed88c035-70ea-4b36-8401-22930e6c2ea5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:58.881755 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed88c035-70ea-4b36-8401-22930e6c2ea5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:58.882541 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed88c035-70ea-4b36-8401-22930e6c2ea5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:58.885960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed88c035-70ea-4b36-8401-22930e6c2ea5 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:58.903142 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 829/1678] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:58 2022] GET /identity/v3/roles => generated 1420 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:58.907153 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed88c035-70ea-4b36-8401-22930e6c2ea5 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:58.923732 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed88c035-70ea-4b36-8401-22930e6c2ea5 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:58.931986 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0455e3c1-bda4-49a7-b81a-257042e816bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:58.932962 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0455e3c1-bda4-49a7-b81a-257042e816bd admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:58.933361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0455e3c1-bda4-49a7-b81a-257042e816bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:58.933750 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0455e3c1-bda4-49a7-b81a-257042e816bd admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:58.935035 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0455e3c1-bda4-49a7-b81a-257042e816bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:58.937384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0455e3c1-bda4-49a7-b81a-257042e816bd admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:58.953137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 830/1679] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:58 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:56:58.957370 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0455e3c1-bda4-49a7-b81a-257042e816bd admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:58.977394 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8702b94b-4598-4688-86ca-d5cad9b691ca admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:58.986145 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0455e3c1-bda4-49a7-b81a-257042e816bd admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:58.989644 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c99d50-a147-4e8e-a01e-bd314bc212fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:58.990445 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c99d50-a147-4e8e-a01e-bd314bc212fd admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:58.990725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c99d50-a147-4e8e-a01e-bd314bc212fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:58.990998 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c99d50-a147-4e8e-a01e-bd314bc212fd admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:58.991752 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78c99d50-a147-4e8e-a01e-bd314bc212fd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:58.993989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78c99d50-a147-4e8e-a01e-bd314bc212fd admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:59.004657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb26cd2-2bc3-4a5d-9747-6aa3c31ae6e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:59.006151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb26cd2-2bc3-4a5d-9747-6aa3c31ae6e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:59.006579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb26cd2-2bc3-4a5d-9747-6aa3c31ae6e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:59.009046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb26cd2-2bc3-4a5d-9747-6aa3c31ae6e4 admin admin] PATH_INFO: `/v3/projects/6177467bc9c4421d83fc6127566c6190/users/b717073926694b16a642eaf46062c0b1/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:59.046305 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 850/1680] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:56:58 2022] POST /identity/v3/users => generated 279 bytes in 311 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:59.047545 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c99d50-a147-4e8e-a01e-bd314bc212fd admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:59.055058 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb26cd2-2bc3-4a5d-9747-6aa3c31ae6e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6177467bc9c4421d83fc6127566c6190, user_id=b717073926694b16a642eaf46062c0b1, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:59.062394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb26cd2-2bc3-4a5d-9747-6aa3c31ae6e4 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:59.069042 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c99d50-a147-4e8e-a01e-bd314bc212fd admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:59.079611 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc75fc63-9d19-4ad8-9f84-785e7c98cc0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:59.080220 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc75fc63-9d19-4ad8-9f84-785e7c98cc0f admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:59.080508 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc75fc63-9d19-4ad8-9f84-785e7c98cc0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:59.081841 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc75fc63-9d19-4ad8-9f84-785e7c98cc0f admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:59.083246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc75fc63-9d19-4ad8-9f84-785e7c98cc0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:59.089740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc75fc63-9d19-4ad8-9f84-785e7c98cc0f admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:59.112440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 851/1681] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:59 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:56:59.113845 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc75fc63-9d19-4ad8-9f84-785e7c98cc0f admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:59.117654 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 831/1682] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:56:58 2022] PUT /identity/v3/projects/6177467bc9c4421d83fc6127566c6190/users/b717073926694b16a642eaf46062c0b1/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:56:59.129675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6573538-a1aa-4be8-87aa-8eb4e0afd416 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:59.131112 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6573538-a1aa-4be8-87aa-8eb4e0afd416 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:59.132052 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6573538-a1aa-4be8-87aa-8eb4e0afd416 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:59.140104 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc75fc63-9d19-4ad8-9f84-785e7c98cc0f admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:59.228595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-d6573538-a1aa-4be8-87aa-8eb4e0afd416 None None] MFA Rules not processed for user `b717073926694b16a642eaf46062c0b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:59.230226 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d6573538-a1aa-4be8-87aa-8eb4e0afd416 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:59.274898 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485ff53a-3e4e-40c1-b925-78e41f40d062 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:59.276645 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485ff53a-3e4e-40c1-b925-78e41f40d062 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:59.277738 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485ff53a-3e4e-40c1-b925-78e41f40d062 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:59.278186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485ff53a-3e4e-40c1-b925-78e41f40d062 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:59.279087 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485ff53a-3e4e-40c1-b925-78e41f40d062 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:59.280959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485ff53a-3e4e-40c1-b925-78e41f40d062 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:59.321139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 852/1683] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:56:59 2022] GET /identity/v3/roles => generated 1420 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:56:59.321863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485ff53a-3e4e-40c1-b925-78e41f40d062 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:59.345048 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485ff53a-3e4e-40c1-b925-78e41f40d062 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:59.366226 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7fb802-5447-4af1-81e4-dd1bf0d3484c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:59.367415 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7fb802-5447-4af1-81e4-dd1bf0d3484c admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:59.367415 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7fb802-5447-4af1-81e4-dd1bf0d3484c admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:59.367585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7fb802-5447-4af1-81e4-dd1bf0d3484c admin admin] PATH_INFO: `/v3/projects/0ca870add6214f1394474d6201aef62b/users/5eb3ddf6498e4f0c9375c53ba0ed6be5/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:59.404748 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7fb802-5447-4af1-81e4-dd1bf0d3484c admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ca870add6214f1394474d6201aef62b, user_id=5eb3ddf6498e4f0c9375c53ba0ed6be5, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:59.409032 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7fb802-5447-4af1-81e4-dd1bf0d3484c admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:59.437815 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 832/1684] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:59 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 305 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:59.438496 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6573538-a1aa-4be8-87aa-8eb4e0afd416 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:59.456900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 853/1685] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:56:59 2022] PUT /identity/v3/projects/0ca870add6214f1394474d6201aef62b/users/5eb3ddf6498e4f0c9375c53ba0ed6be5/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:56:59.467543 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6573538-a1aa-4be8-87aa-8eb4e0afd416 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:59.471955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793fdb86-a16c-4e22-a1d7-272dba5af760 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:59.478445 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793fdb86-a16c-4e22-a1d7-272dba5af760 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:59.478825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793fdb86-a16c-4e22-a1d7-272dba5af760 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:59.492397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:59.545509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-793fdb86-a16c-4e22-a1d7-272dba5af760 None None] MFA Rules not processed for user `5eb3ddf6498e4f0c9375c53ba0ed6be5`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:56:59.547151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-793fdb86-a16c-4e22-a1d7-272dba5af760 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:56:59.561140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:56:59.608606 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec33893-a06b-42ed-9e17-c77b62c7eb70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:59.609260 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec33893-a06b-42ed-9e17-c77b62c7eb70 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:59.609542 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec33893-a06b-42ed-9e17-c77b62c7eb70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:59.609813 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec33893-a06b-42ed-9e17-c77b62c7eb70 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:59.610510 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec33893-a06b-42ed-9e17-c77b62c7eb70 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:59.612539 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec33893-a06b-42ed-9e17-c77b62c7eb70 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:59.682695 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 833/1686] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:56:59 2022] POST /identity/v3/projects => generated 385 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:56:59.687210 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec33893-a06b-42ed-9e17-c77b62c7eb70 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:59.708425 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 854/1687] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:56:59 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:56:59.719761 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793fdb86-a16c-4e22-a1d7-272dba5af760 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:56:59.738241 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793fdb86-a16c-4e22-a1d7-272dba5af760 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:59.955063 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec33893-a06b-42ed-9e17-c77b62c7eb70 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:56:59.978157 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83455d7b-7f7b-46f5-84ea-55d2e405dc87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:59.979273 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83455d7b-7f7b-46f5-84ea-55d2e405dc87 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:56:59.979769 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83455d7b-7f7b-46f5-84ea-55d2e405dc87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:56:59.980273 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83455d7b-7f7b-46f5-84ea-55d2e405dc87 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:56:59.981469 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83455d7b-7f7b-46f5-84ea-55d2e405dc87 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:56:59.985343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83455d7b-7f7b-46f5-84ea-55d2e405dc87 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:56:59.998830 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cdbef9-3d2e-4257-b37e-da202e5ef9b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:56:59.999913 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cdbef9-3d2e-4257-b37e-da202e5ef9b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:00.000523 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cdbef9-3d2e-4257-b37e-da202e5ef9b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:00.000948 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cdbef9-3d2e-4257-b37e-da202e5ef9b9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:00.001960 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cdbef9-3d2e-4257-b37e-da202e5ef9b9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:00.004276 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cdbef9-3d2e-4257-b37e-da202e5ef9b9 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:00.064567 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 834/1688] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:56:59 2022] POST /identity/v3/users => generated 299 bytes in 375 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:57:00.064783 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 855/1689] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:56:59 2022] POST /identity/v3/projects => generated 357 bytes in 344 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:57:00.067456 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83455d7b-7f7b-46f5-84ea-55d2e405dc87 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:00.067647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cdbef9-3d2e-4257-b37e-da202e5ef9b9 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:00.087003 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83455d7b-7f7b-46f5-84ea-55d2e405dc87 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:00.226873 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff4086f-c001-417c-bdff-b791de91eb27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:00.227554 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff4086f-c001-417c-bdff-b791de91eb27 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:00.227843 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff4086f-c001-417c-bdff-b791de91eb27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:00.228125 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff4086f-c001-417c-bdff-b791de91eb27 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:00.228809 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff4086f-c001-417c-bdff-b791de91eb27 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:00.230560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff4086f-c001-417c-bdff-b791de91eb27 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:00.241728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cdbef9-3d2e-4257-b37e-da202e5ef9b9 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:00.245922 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 835/1690] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:00 2022] GET /identity/v3/roles => generated 1420 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:57:00.247196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff4086f-c001-417c-bdff-b791de91eb27 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:00.254392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4fc2b5-55d8-4e6b-b790-960d55cf1859 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:00.255071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4fc2b5-55d8-4e6b-b790-960d55cf1859 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:00.255356 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4fc2b5-55d8-4e6b-b790-960d55cf1859 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:00.255689 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4fc2b5-55d8-4e6b-b790-960d55cf1859 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:00.256535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4fc2b5-55d8-4e6b-b790-960d55cf1859 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:00.258669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4fc2b5-55d8-4e6b-b790-960d55cf1859 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:00.274331 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff4086f-c001-417c-bdff-b791de91eb27 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:00.281560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f799fa85-6661-416e-a4ce-d701ca2e02c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:00.282313 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f799fa85-6661-416e-a4ce-d701ca2e02c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:00.282655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f799fa85-6661-416e-a4ce-d701ca2e02c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:00.282934 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f799fa85-6661-416e-a4ce-d701ca2e02c6 admin admin] PATH_INFO: `/v3/projects/cb5cbf3c334f4de5ae0377e60cdc8358/users/a49dd9720f684ff6983ed141c56d4449/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:00.315170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f799fa85-6661-416e-a4ce-d701ca2e02c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb5cbf3c334f4de5ae0377e60cdc8358, user_id=a49dd9720f684ff6983ed141c56d4449, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:00.321938 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f799fa85-6661-416e-a4ce-d701ca2e02c6 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:00.333565 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:00.339483 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 856/1691] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:57:00 2022] POST /identity/v3/users => generated 285 bytes in 272 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:57:00.345754 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4fc2b5-55d8-4e6b-b790-960d55cf1859 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:00.373161 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 836/1692] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:57:00 2022] PUT /identity/v3/projects/cb5cbf3c334f4de5ae0377e60cdc8358/users/a49dd9720f684ff6983ed141c56d4449/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:00.375255 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4fc2b5-55d8-4e6b-b790-960d55cf1859 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:00.376498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f799fa85-6661-416e-a4ce-d701ca2e02c6 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:00.454578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:00.524189 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f799fa85-6661-416e-a4ce-d701ca2e02c6 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:00.549980 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61de0b52-e2dc-40ca-bd1c-0d8b6b873bb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:00.551380 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61de0b52-e2dc-40ca-bd1c-0d8b6b873bb4 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:00.551739 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61de0b52-e2dc-40ca-bd1c-0d8b6b873bb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:00.552035 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61de0b52-e2dc-40ca-bd1c-0d8b6b873bb4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:00.553531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61de0b52-e2dc-40ca-bd1c-0d8b6b873bb4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:00.567374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61de0b52-e2dc-40ca-bd1c-0d8b6b873bb4 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:00.588014 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 837/1693] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:00 2022] GET /identity/v3/roles => generated 1420 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:57:00.588365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a2a7aa-2a9e-4cd2-907f-3fd3c5e3f0c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:00.588920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a2a7aa-2a9e-4cd2-907f-3fd3c5e3f0c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:00.589132 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a2a7aa-2a9e-4cd2-907f-3fd3c5e3f0c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:00.589438 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a2a7aa-2a9e-4cd2-907f-3fd3c5e3f0c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:00.590113 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a2a7aa-2a9e-4cd2-907f-3fd3c5e3f0c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:00.591873 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61de0b52-e2dc-40ca-bd1c-0d8b6b873bb4 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:00.596091 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a2a7aa-2a9e-4cd2-907f-3fd3c5e3f0c8 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:00.622976 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61de0b52-e2dc-40ca-bd1c-0d8b6b873bb4 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:00.634498 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 857/1694] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:00 2022] GET /identity/v3/roles => generated 1420 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:57:00.635581 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a2a7aa-2a9e-4cd2-907f-3fd3c5e3f0c8 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:00.663162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a2a7aa-2a9e-4cd2-907f-3fd3c5e3f0c8 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:00.665279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfd0f5c-7236-46f1-80c5-4c0af8383859 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:00.665971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfd0f5c-7236-46f1-80c5-4c0af8383859 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:00.666250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfd0f5c-7236-46f1-80c5-4c0af8383859 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:00.666550 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfd0f5c-7236-46f1-80c5-4c0af8383859 admin admin] PATH_INFO: `/v3/projects/cb5cbf3c334f4de5ae0377e60cdc8358/users/a49dd9720f684ff6983ed141c56d4449/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:00.669300 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dfd0f5c-7236-46f1-80c5-4c0af8383859 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb5cbf3c334f4de5ae0377e60cdc8358, user_id=a49dd9720f684ff6983ed141c56d4449, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:00.672006 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dfd0f5c-7236-46f1-80c5-4c0af8383859 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:00.672167 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1857a732-9647-4857-9706-e2173b8a0dcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:00.672794 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1857a732-9647-4857-9706-e2173b8a0dcb admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:00.673078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1857a732-9647-4857-9706-e2173b8a0dcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:00.673355 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1857a732-9647-4857-9706-e2173b8a0dcb admin admin] PATH_INFO: `/v3/projects/51fde31499574e58a2d0d72ebb521a68/users/180e810b87724505821317da0f4cff7a/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:00.716836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 838/1695] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:57:00 2022] PUT /identity/v3/projects/cb5cbf3c334f4de5ae0377e60cdc8358/users/a49dd9720f684ff6983ed141c56d4449/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:00.720334 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e6503b-6402-4903-a6f5-76af98aa4052 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:00.720649 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e6503b-6402-4903-a6f5-76af98aa4052 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:00.720927 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e6503b-6402-4903-a6f5-76af98aa4052 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:00.760047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1857a732-9647-4857-9706-e2173b8a0dcb admin admin] RBAC: Authorizing `identity:create_grant(project_id=51fde31499574e58a2d0d72ebb521a68, user_id=180e810b87724505821317da0f4cff7a, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:00.760047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1857a732-9647-4857-9706-e2173b8a0dcb admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:00.800999 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-35e6503b-6402-4903-a6f5-76af98aa4052 None None] MFA Rules not processed for user `a49dd9720f684ff6983ed141c56d4449`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:00.804060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-35e6503b-6402-4903-a6f5-76af98aa4052 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:00.810450 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 858/1696] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:57:00 2022] PUT /identity/v3/projects/51fde31499574e58a2d0d72ebb521a68/users/180e810b87724505821317da0f4cff7a/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:00.822188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1857a732-9647-4857-9706-e2173b8a0dcb admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:01.060281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1857a732-9647-4857-9706-e2173b8a0dcb admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:01.066390 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 839/1697] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:00 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 349 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:57:01.084334 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e6503b-6402-4903-a6f5-76af98aa4052 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:01.085934 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-35e6503b-6402-4903-a6f5-76af98aa4052 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:01.103060 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb11d0f-c33a-4734-bc38-f0680974bae2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:01.103726 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb11d0f-c33a-4734-bc38-f0680974bae2 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:01.104009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb11d0f-c33a-4734-bc38-f0680974bae2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:01.104286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb11d0f-c33a-4734-bc38-f0680974bae2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:01.104949 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb11d0f-c33a-4734-bc38-f0680974bae2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:01.106630 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb11d0f-c33a-4734-bc38-f0680974bae2 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:01.154527 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 859/1698] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:00 2022] GET /identity/v3/roles => generated 1420 bytes in 333 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:57:01.165634 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb11d0f-c33a-4734-bc38-f0680974bae2 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:01.205914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb11d0f-c33a-4734-bc38-f0680974bae2 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:01.223427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e4f45e-604e-4f74-94e8-b3bfb1aa57fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:01.224376 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e4f45e-604e-4f74-94e8-b3bfb1aa57fd admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:01.224609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e4f45e-604e-4f74-94e8-b3bfb1aa57fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:01.224894 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e4f45e-604e-4f74-94e8-b3bfb1aa57fd admin admin] PATH_INFO: `/v3/projects/51fde31499574e58a2d0d72ebb521a68/users/180e810b87724505821317da0f4cff7a/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:01.227269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e4f45e-604e-4f74-94e8-b3bfb1aa57fd admin admin] RBAC: Authorizing `identity:create_grant(project_id=51fde31499574e58a2d0d72ebb521a68, user_id=180e810b87724505821317da0f4cff7a, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:01.230161 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e4f45e-604e-4f74-94e8-b3bfb1aa57fd admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:01.268178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e6503b-6402-4903-a6f5-76af98aa4052 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:01.279205 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 860/1699] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:57:01 2022] PUT /identity/v3/projects/51fde31499574e58a2d0d72ebb521a68/users/180e810b87724505821317da0f4cff7a/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:01.280786 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55714b58-bd49-4a44-9a62-b6e23932a1fb None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:01.281059 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55714b58-bd49-4a44-9a62-b6e23932a1fb None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:01.281323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55714b58-bd49-4a44-9a62-b6e23932a1fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:01.340447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-55714b58-bd49-4a44-9a62-b6e23932a1fb None None] MFA Rules not processed for user `180e810b87724505821317da0f4cff7a`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:01.342193 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:01.342847 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-55714b58-bd49-4a44-9a62-b6e23932a1fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:01.438245 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457fff55-2eaf-4835-b413-ed2bb792b6a2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:01.441716 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457fff55-2eaf-4835-b413-ed2bb792b6a2 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:01.442420 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457fff55-2eaf-4835-b413-ed2bb792b6a2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:01.443223 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457fff55-2eaf-4835-b413-ed2bb792b6a2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:01.526930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 861/1700] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:01 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 247 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:57:01.527347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:01.530056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c04ce7-3366-4a09-a692-71b0148a0f49 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:01.530353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c04ce7-3366-4a09-a692-71b0148a0f49 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:01.531402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c04ce7-3366-4a09-a692-71b0148a0f49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:01.578203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-95c04ce7-3366-4a09-a692-71b0148a0f49 None None] MFA Rules not processed for user `468d75d910874fa8bb0f4dacb589f64c`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:01.579372 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-95c04ce7-3366-4a09-a692-71b0148a0f49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:01.608660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457fff55-2eaf-4835-b413-ed2bb792b6a2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:01.614569 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457fff55-2eaf-4835-b413-ed2bb792b6a2 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:01.635892 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 840/1701] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:01 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 552 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:57:01.637260 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457fff55-2eaf-4835-b413-ed2bb792b6a2 service cinder] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:01.640882 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-457fff55-2eaf-4835-b413-ed2bb792b6a2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:01.680686 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457fff55-2eaf-4835-b413-ed2bb792b6a2 service cinder] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:01.689466 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d75b88f-bee8-40c4-9d48-d3a1678e69c0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:01.691174 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d75b88f-bee8-40c4-9d48-d3a1678e69c0 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:01.691643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d75b88f-bee8-40c4-9d48-d3a1678e69c0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:01.692061 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d75b88f-bee8-40c4-9d48-d3a1678e69c0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:01.735833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 862/1702] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:01 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:01.743648 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d75b88f-bee8-40c4-9d48-d3a1678e69c0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:01.747260 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f776cbfa-b875-421f-8a1d-c1ab4554fdb6 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:01.747978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f776cbfa-b875-421f-8a1d-c1ab4554fdb6 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:01.748731 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f776cbfa-b875-421f-8a1d-c1ab4554fdb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:01.748947 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d75b88f-bee8-40c4-9d48-d3a1678e69c0 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:01.766130 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 841/1703] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:01 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:01.767982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d75b88f-bee8-40c4-9d48-d3a1678e69c0 service cinder] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:01.826344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-f776cbfa-b875-421f-8a1d-c1ab4554fdb6 None None] MFA Rules not processed for user `468d75d910874fa8bb0f4dacb589f64c`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:01.835315 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-f776cbfa-b875-421f-8a1d-c1ab4554fdb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:01.874664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 863/1704] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:01 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:01.877393 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f776cbfa-b875-421f-8a1d-c1ab4554fdb6 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:01.884777 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-f776cbfa-b875-421f-8a1d-c1ab4554fdb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:01.979982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d75b88f-bee8-40c4-9d48-d3a1678e69c0 service cinder] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:01.991545 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14298424-311b-43fc-b59a-45a858722b09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:01.992569 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14298424-311b-43fc-b59a-45a858722b09 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:01.992814 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14298424-311b-43fc-b59a-45a858722b09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:01.993164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14298424-311b-43fc-b59a-45a858722b09 admin admin] PATH_INFO: `/v3/users/b717073926694b16a642eaf46062c0b1` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:01.995187 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14298424-311b-43fc-b59a-45a858722b09 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b717073926694b16a642eaf46062c0b1)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:02.002767 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14298424-311b-43fc-b59a-45a858722b09 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:02.127997 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-14298424-311b-43fc-b59a-45a858722b09 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b717073926694b16a642eaf46062c0b1'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:02.183976 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-14298424-311b-43fc-b59a-45a858722b09 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b717073926694b16a642eaf46062c0b1'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:02.186020 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-14298424-311b-43fc-b59a-45a858722b09 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b717073926694b16a642eaf46062c0b1'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:02.186696 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-14298424-311b-43fc-b59a-45a858722b09 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b717073926694b16a642eaf46062c0b1'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:02.214695 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:02.228784 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f776cbfa-b875-421f-8a1d-c1ab4554fdb6 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:02.252421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-14298424-311b-43fc-b59a-45a858722b09 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b717073926694b16a642eaf46062c0b1'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:02.265275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d125d1c-1c41-4f53-8296-7ed6b13cc878 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:02.266256 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d125d1c-1c41-4f53-8296-7ed6b13cc878 service nova] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:02.267090 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d125d1c-1c41-4f53-8296-7ed6b13cc878 service nova] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:02.267564 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d125d1c-1c41-4f53-8296-7ed6b13cc878 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:02.278100 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-8d125d1c-1c41-4f53-8296-7ed6b13cc878 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:02.367324 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d125d1c-1c41-4f53-8296-7ed6b13cc878 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:02.375267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 842/1705] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:57:01 2022] DELETE /identity/v3/users/b717073926694b16a642eaf46062c0b1 => generated 0 bytes in 607 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:02.375974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d125d1c-1c41-4f53-8296-7ed6b13cc878 service nova] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:02.376982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14298424-311b-43fc-b59a-45a858722b09 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:02.382640 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-14298424-311b-43fc-b59a-45a858722b09 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:02.418794 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 864/1706] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:01 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 542 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:02.422629 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54ce7c9-485d-4ffd-9b31-000bc506a91d None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:02.423096 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54ce7c9-485d-4ffd-9b31-000bc506a91d None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:02.423539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54ce7c9-485d-4ffd-9b31-000bc506a91d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:02.512497 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-d54ce7c9-485d-4ffd-9b31-000bc506a91d None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:02.513728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-d54ce7c9-485d-4ffd-9b31-000bc506a91d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:02.610215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14298424-311b-43fc-b59a-45a858722b09 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:02.624271 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9332738a-be83-415d-990b-a56469c31521 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:02.631829 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9332738a-be83-415d-990b-a56469c31521 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:02.632637 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9332738a-be83-415d-990b-a56469c31521 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:02.633409 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9332738a-be83-415d-990b-a56469c31521 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:02.642090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-9332738a-be83-415d-990b-a56469c31521 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:02.749407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 865/1707] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 330 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:02.802870 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54ce7c9-485d-4ffd-9b31-000bc506a91d None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:02.804451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-d54ce7c9-485d-4ffd-9b31-000bc506a91d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:02.824937 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:02.935207 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54ce7c9-485d-4ffd-9b31-000bc506a91d None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:02.952805 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174d2b12-5bd8-47b9-8aef-f0bf305dd880 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:02.953443 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174d2b12-5bd8-47b9-8aef-f0bf305dd880 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:02.953756 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174d2b12-5bd8-47b9-8aef-f0bf305dd880 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:02.953980 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174d2b12-5bd8-47b9-8aef-f0bf305dd880 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:02.981102 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174d2b12-5bd8-47b9-8aef-f0bf305dd880 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:02.981838 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9332738a-be83-415d-990b-a56469c31521 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:02.982786 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174d2b12-5bd8-47b9-8aef-f0bf305dd880 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:02.993678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9332738a-be83-415d-990b-a56469c31521 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:03.004044 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 866/1708] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:57:03.071853 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 843/1709] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:02 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 696 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:03.496058 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3968ea4-cd6e-4975-be8b-4a8bec3227b6 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:03.496604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3968ea4-cd6e-4975-be8b-4a8bec3227b6 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:03.497031 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3968ea4-cd6e-4975-be8b-4a8bec3227b6 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:03.498677 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 867/1710] 10.222.0.42 () {54 vars in 912 bytes} [Sun Sep 18 00:57:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:57:03.511470 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb6c07b-336e-4154-b5d1-d82739dd7f8b None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:03.511974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb6c07b-336e-4154-b5d1-d82739dd7f8b None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:03.512467 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb6c07b-336e-4154-b5d1-d82739dd7f8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:03.666254 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-6cb6c07b-336e-4154-b5d1-d82739dd7f8b None None] MFA Rules not processed for user `9b0510812b60454586a781e6399e77a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:03.668493 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-6cb6c07b-336e-4154-b5d1-d82739dd7f8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:03.702700 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 844/1711] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 18 00:57:03 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:03.730573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3968ea4-cd6e-4975-be8b-4a8bec3227b6 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:03.732290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-f3968ea4-cd6e-4975-be8b-4a8bec3227b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:03.758259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3968ea4-cd6e-4975-be8b-4a8bec3227b6 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:03.769197 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef13076e-9bb2-48a7-af89-4edba6e1613a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:03.773219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef13076e-9bb2-48a7-af89-4edba6e1613a service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:03.773781 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef13076e-9bb2-48a7-af89-4edba6e1613a service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:03.774207 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef13076e-9bb2-48a7-af89-4edba6e1613a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:03.791316 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef13076e-9bb2-48a7-af89-4edba6e1613a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:03.793232 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef13076e-9bb2-48a7-af89-4edba6e1613a service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:03.830211 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 868/1712] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:03 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:05.267566 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8cc1e3-218f-4212-a480-6ec8332fd95e None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:05.268086 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8cc1e3-218f-4212-a480-6ec8332fd95e None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:05.268165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8cc1e3-218f-4212-a480-6ec8332fd95e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:05.298813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef13076e-9bb2-48a7-af89-4edba6e1613a service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:05.300447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-ef13076e-9bb2-48a7-af89-4edba6e1613a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:05.339538 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-5f8cc1e3-218f-4212-a480-6ec8332fd95e None None] MFA Rules not processed for user `54d4e437036640b58fda503c4d213f5c`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:05.340795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-5f8cc1e3-218f-4212-a480-6ec8332fd95e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:05.341303 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef13076e-9bb2-48a7-af89-4edba6e1613a service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:05.378262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53b859c-3059-4ff4-b714-b9bd77aa44d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:05.380181 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53b859c-3059-4ff4-b714-b9bd77aa44d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:05.382272 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53b859c-3059-4ff4-b714-b9bd77aa44d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:05.382726 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53b859c-3059-4ff4-b714-b9bd77aa44d6 admin admin] PATH_INFO: `/v3/projects/6177467bc9c4421d83fc6127566c6190` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:05.384210 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53b859c-3059-4ff4-b714-b9bd77aa44d6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6177467bc9c4421d83fc6127566c6190)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:05.387177 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53b859c-3059-4ff4-b714-b9bd77aa44d6 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:05.535379 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d53b859c-3059-4ff4-b714-b9bd77aa44d6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6177467bc9c4421d83fc6127566c6190', 'request_id': 'req-d53b859c-3059-4ff4-b714-b9bd77aa44d6'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:05.550604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d53b859c-3059-4ff4-b714-b9bd77aa44d6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6177467bc9c4421d83fc6127566c6190', 'request_id': 'req-d53b859c-3059-4ff4-b714-b9bd77aa44d6'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:05.582097 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d53b859c-3059-4ff4-b714-b9bd77aa44d6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6177467bc9c4421d83fc6127566c6190', 'request_id': 'req-d53b859c-3059-4ff4-b714-b9bd77aa44d6'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:05.586868 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:05.603780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 845/1713] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:05 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 337 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:05.617330 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceba31a0-4b2c-48d7-8593-d9b343863b47 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:05.617803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceba31a0-4b2c-48d7-8593-d9b343863b47 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:05.618195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceba31a0-4b2c-48d7-8593-d9b343863b47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:05.648986 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d53b859c-3059-4ff4-b714-b9bd77aa44d6 admin admin] The token cache is being invalidate because project 6177467bc9c4421d83fc6127566c6190 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:57:05.649274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d53b859c-3059-4ff4-b714-b9bd77aa44d6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:05.653317 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d53b859c-3059-4ff4-b714-b9bd77aa44d6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:05.654738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-ceba31a0-4b2c-48d7-8593-d9b343863b47 None None] MFA Rules not processed for user `54d4e437036640b58fda503c4d213f5c`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:05.654999 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 869/1714] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:57:05 2022] DELETE /identity/v3/projects/6177467bc9c4421d83fc6127566c6190 => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:57:05.657166 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ceba31a0-4b2c-48d7-8593-d9b343863b47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:05.658898 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53b859c-3059-4ff4-b714-b9bd77aa44d6 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:05.663522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-d53b859c-3059-4ff4-b714-b9bd77aa44d6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:05.824641 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 846/1715] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:05 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:05.845194 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceba31a0-4b2c-48d7-8593-d9b343863b47 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:05.853906 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ceba31a0-4b2c-48d7-8593-d9b343863b47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:05.932346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53b859c-3059-4ff4-b714-b9bd77aa44d6 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:05.947570 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c917faf-cd3e-4850-813a-737655824149 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:05.948563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c917faf-cd3e-4850-813a-737655824149 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:05.949002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c917faf-cd3e-4850-813a-737655824149 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:05.956528 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c917faf-cd3e-4850-813a-737655824149 admin admin] PATH_INFO: `/v3/users/a49dd9720f684ff6983ed141c56d4449` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:05.958136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c917faf-cd3e-4850-813a-737655824149 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a49dd9720f684ff6983ed141c56d4449)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:05.967910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c917faf-cd3e-4850-813a-737655824149 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:06.064504 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:06.163034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5c917faf-cd3e-4850-813a-737655824149 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a49dd9720f684ff6983ed141c56d4449'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:06.198721 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5c917faf-cd3e-4850-813a-737655824149 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a49dd9720f684ff6983ed141c56d4449'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:06.229944 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5c917faf-cd3e-4850-813a-737655824149 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a49dd9720f684ff6983ed141c56d4449'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:06.230780 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5c917faf-cd3e-4850-813a-737655824149 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a49dd9720f684ff6983ed141c56d4449'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:06.249165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceba31a0-4b2c-48d7-8593-d9b343863b47 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:06.252600 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5c917faf-cd3e-4850-813a-737655824149 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a49dd9720f684ff6983ed141c56d4449'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:06.267136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 870/1716] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:57:05 2022] DELETE /identity/v3/users/a49dd9720f684ff6983ed141c56d4449 => generated 0 bytes in 606 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:06.267136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c917faf-cd3e-4850-813a-737655824149 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:06.270335 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472b7d2a-72f0-4dc3-a63f-83402a266aa5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:06.270880 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-5c917faf-cd3e-4850-813a-737655824149 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:06.272627 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472b7d2a-72f0-4dc3-a63f-83402a266aa5 service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:06.273136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472b7d2a-72f0-4dc3-a63f-83402a266aa5 service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:06.273614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472b7d2a-72f0-4dc3-a63f-83402a266aa5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:06.280081 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-472b7d2a-72f0-4dc3-a63f-83402a266aa5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:06.325710 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472b7d2a-72f0-4dc3-a63f-83402a266aa5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:06.328361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472b7d2a-72f0-4dc3-a63f-83402a266aa5 service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:06.379350 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 847/1717] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:05 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 534 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:06.381369 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472b7d2a-72f0-4dc3-a63f-83402a266aa5 service nova] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:06.385390 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-472b7d2a-72f0-4dc3-a63f-83402a266aa5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:06.531341 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c917faf-cd3e-4850-813a-737655824149 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:06.558742 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d15f80-dada-4f3e-bc36-1af575d12bc7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8f30116fb5ce43dab69efa8bafd9d049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:06.559703 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d15f80-dada-4f3e-bc36-1af575d12bc7 service placement] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:06.560311 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d15f80-dada-4f3e-bc36-1af575d12bc7 service placement] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:06.560886 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d15f80-dada-4f3e-bc36-1af575d12bc7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:06.566996 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-47d15f80-dada-4f3e-bc36-1af575d12bc7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:06.605349 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472b7d2a-72f0-4dc3-a63f-83402a266aa5 service nova] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:06.622556 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d15f80-dada-4f3e-bc36-1af575d12bc7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:06.624569 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d15f80-dada-4f3e-bc36-1af575d12bc7 service placement] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:06.662133 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 871/1718] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:06 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:06.745452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:06.773923 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b9b094-c87b-47e9-ad05-e1df306d5beb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:06.774782 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b9b094-c87b-47e9-ad05-e1df306d5beb service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:06.775169 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b9b094-c87b-47e9-ad05-e1df306d5beb service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:06.775551 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b9b094-c87b-47e9-ad05-e1df306d5beb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:06.778340 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-63b9b094-c87b-47e9-ad05-e1df306d5beb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:07.078091 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b9b094-c87b-47e9-ad05-e1df306d5beb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:07.080688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b9b094-c87b-47e9-ad05-e1df306d5beb service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:07.117442 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 848/1719] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:06 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 737 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:08.405321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d15f80-dada-4f3e-bc36-1af575d12bc7 service placement] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:08.407730 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-47d15f80-dada-4f3e-bc36-1af575d12bc7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:08.726916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d15f80-dada-4f3e-bc36-1af575d12bc7 service placement] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:08.748785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19ed6c3-c303-4b8c-81e4-fc66e79e546e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:08.750297 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19ed6c3-c303-4b8c-81e4-fc66e79e546e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:08.760750 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19ed6c3-c303-4b8c-81e4-fc66e79e546e admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:08.761320 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19ed6c3-c303-4b8c-81e4-fc66e79e546e admin admin] PATH_INFO: `/v3/projects/cb5cbf3c334f4de5ae0377e60cdc8358` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:08.765652 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19ed6c3-c303-4b8c-81e4-fc66e79e546e admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb5cbf3c334f4de5ae0377e60cdc8358)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:08.779592 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19ed6c3-c303-4b8c-81e4-fc66e79e546e admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:08.927764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:08.962878 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d19ed6c3-c303-4b8c-81e4-fc66e79e546e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb5cbf3c334f4de5ae0377e60cdc8358', 'request_id': 'req-d19ed6c3-c303-4b8c-81e4-fc66e79e546e'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:08.979521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d19ed6c3-c303-4b8c-81e4-fc66e79e546e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb5cbf3c334f4de5ae0377e60cdc8358', 'request_id': 'req-d19ed6c3-c303-4b8c-81e4-fc66e79e546e'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:08.991087 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d19ed6c3-c303-4b8c-81e4-fc66e79e546e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb5cbf3c334f4de5ae0377e60cdc8358', 'request_id': 'req-d19ed6c3-c303-4b8c-81e4-fc66e79e546e'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:09.010234 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d19ed6c3-c303-4b8c-81e4-fc66e79e546e admin admin] The token cache is being invalidate because project cb5cbf3c334f4de5ae0377e60cdc8358 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:57:09.011134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d19ed6c3-c303-4b8c-81e4-fc66e79e546e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:09.011870 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d19ed6c3-c303-4b8c-81e4-fc66e79e546e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:09.016682 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 872/1720] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:57:08 2022] DELETE /identity/v3/projects/cb5cbf3c334f4de5ae0377e60cdc8358 => generated 0 bytes in 612 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:09.022744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d9de6c-ea5c-471f-8dce-91af77addbb2 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:09.023238 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d9de6c-ea5c-471f-8dce-91af77addbb2 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:09.023316 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d9de6c-ea5c-471f-8dce-91af77addbb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:09.055327 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-d7d9de6c-ea5c-471f-8dce-91af77addbb2 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:09.067237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d7d9de6c-ea5c-471f-8dce-91af77addbb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:09.223157 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6febceae-d7f7-4d18-abd4-863e943c4964 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:09.223449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6febceae-d7f7-4d18-abd4-863e943c4964 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:09.223725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6febceae-d7f7-4d18-abd4-863e943c4964 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:09.234888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 849/1721] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:09.280154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-6febceae-d7f7-4d18-abd4-863e943c4964 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:09.282570 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-6febceae-d7f7-4d18-abd4-863e943c4964 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:09.308335 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 873/1722] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:09.312622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d9de6c-ea5c-471f-8dce-91af77addbb2 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:09.329976 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d9de6c-ea5c-471f-8dce-91af77addbb2 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:09.353425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e0cf2b-8c1c-49e8-9983-9fc877c45e95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:09.354091 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e0cf2b-8c1c-49e8-9983-9fc877c45e95 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:09.354366 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e0cf2b-8c1c-49e8-9983-9fc877c45e95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:09.354674 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e0cf2b-8c1c-49e8-9983-9fc877c45e95 admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:09.355336 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e0cf2b-8c1c-49e8-9983-9fc877c45e95 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:57:09.355632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e0cf2b-8c1c-49e8-9983-9fc877c45e95 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:09.357280 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e0cf2b-8c1c-49e8-9983-9fc877c45e95 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:09.398486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6febceae-d7f7-4d18-abd4-863e943c4964 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:09.398927 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 850/1723] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:57:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:57:09.410006 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6febceae-d7f7-4d18-abd4-863e943c4964 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:09.421299 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827d232c-5fd8-4905-9604-b811e7e87c26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:09.423228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827d232c-5fd8-4905-9604-b811e7e87c26 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:09.424302 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827d232c-5fd8-4905-9604-b811e7e87c26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:09.428409 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827d232c-5fd8-4905-9604-b811e7e87c26 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:09.429127 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-827d232c-5fd8-4905-9604-b811e7e87c26 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:09.431135 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-827d232c-5fd8-4905-9604-b811e7e87c26 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:09.500172 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 874/1724] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:57:09 2022] POST /identity/v3/projects => generated 409 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:57:09.503713 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e0cf2b-8c1c-49e8-9983-9fc877c45e95 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:09.537290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e0cf2b-8c1c-49e8-9983-9fc877c45e95 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:09.716027 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4cb8c2-6cc0-49f5-be82-404fcc56f5e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:09.719157 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4cb8c2-6cc0-49f5-be82-404fcc56f5e8 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:09.720880 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4cb8c2-6cc0-49f5-be82-404fcc56f5e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:09.721343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4cb8c2-6cc0-49f5-be82-404fcc56f5e8 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:09.723658 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4cb8c2-6cc0-49f5-be82-404fcc56f5e8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:09.730755 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4cb8c2-6cc0-49f5-be82-404fcc56f5e8 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:09.794933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827d232c-5fd8-4905-9604-b811e7e87c26 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:09.807793 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 851/1725] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:57:09 2022] POST /identity/v3/users => generated 305 bytes in 304 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:57:09.818912 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827d232c-5fd8-4905-9604-b811e7e87c26 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:09.850022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa0f49e-4820-4e3d-87c4-e5f16fe8224c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:09.855050 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa0f49e-4820-4e3d-87c4-e5f16fe8224c admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:09.859766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa0f49e-4820-4e3d-87c4-e5f16fe8224c admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:09.861241 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa0f49e-4820-4e3d-87c4-e5f16fe8224c admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:09.863196 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa0f49e-4820-4e3d-87c4-e5f16fe8224c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:09.870949 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa0f49e-4820-4e3d-87c4-e5f16fe8224c admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:09.939301 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 875/1726] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:09 2022] GET /identity/v3/roles => generated 1420 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:57:09.943526 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4cb8c2-6cc0-49f5-be82-404fcc56f5e8 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:09.973442 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4cb8c2-6cc0-49f5-be82-404fcc56f5e8 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:09.991188 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200517ef-3be0-4e9e-ae90-7f9bc0398b7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:09.992190 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200517ef-3be0-4e9e-ae90-7f9bc0398b7e admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:09.992629 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200517ef-3be0-4e9e-ae90-7f9bc0398b7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:09.993324 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200517ef-3be0-4e9e-ae90-7f9bc0398b7e admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:09.994170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200517ef-3be0-4e9e-ae90-7f9bc0398b7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:09.997523 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200517ef-3be0-4e9e-ae90-7f9bc0398b7e admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:10.044188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d237db-c463-4575-9f1f-f191e74b28da None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:10.044453 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d237db-c463-4575-9f1f-f191e74b28da None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:10.044726 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d237db-c463-4575-9f1f-f191e74b28da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:10.049144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 852/1727] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:09 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:57:10.050112 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200517ef-3be0-4e9e-ae90-7f9bc0398b7e admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:10.081746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200517ef-3be0-4e9e-ae90-7f9bc0398b7e admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:10.113545 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0e18ee-b5f1-40b2-a15d-5aa4a8e0ff61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:10.114235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0e18ee-b5f1-40b2-a15d-5aa4a8e0ff61 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:10.114715 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0e18ee-b5f1-40b2-a15d-5aa4a8e0ff61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:10.115182 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0e18ee-b5f1-40b2-a15d-5aa4a8e0ff61 admin admin] PATH_INFO: `/v3/projects/15684cd8f29440599ea291fd7f9f42a7/users/1a5a1af071224f4988b2f244519a8396/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:10.139011 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:10.191637 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-20d237db-c463-4575-9f1f-f191e74b28da None None] MFA Rules not processed for user `294ae98a8492468893beb3b8c2870d5e`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:10.192888 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-20d237db-c463-4575-9f1f-f191e74b28da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:10.194092 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0e18ee-b5f1-40b2-a15d-5aa4a8e0ff61 admin admin] RBAC: Authorizing `identity:create_grant(project_id=15684cd8f29440599ea291fd7f9f42a7, user_id=1a5a1af071224f4988b2f244519a8396, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:10.204466 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0e18ee-b5f1-40b2-a15d-5aa4a8e0ff61 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:10.236890 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 853/1728] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:57:10 2022] PUT /identity/v3/projects/15684cd8f29440599ea291fd7f9f42a7/users/1a5a1af071224f4988b2f244519a8396/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:10.249846 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7d24cb-8971-4497-aa72-e3b1d47a8125 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:10.250130 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7d24cb-8971-4497-aa72-e3b1d47a8125 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:10.250405 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7d24cb-8971-4497-aa72-e3b1d47a8125 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:10.315236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-ef7d24cb-8971-4497-aa72-e3b1d47a8125 None None] MFA Rules not processed for user `1a5a1af071224f4988b2f244519a8396`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:10.316521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ef7d24cb-8971-4497-aa72-e3b1d47a8125 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:10.400932 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 876/1729] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:10 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 358 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:57:10.414993 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d237db-c463-4575-9f1f-f191e74b28da None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:10.416400 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-20d237db-c463-4575-9f1f-f191e74b28da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:10.635688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 854/1730] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:10 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 387 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:10.639560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7d24cb-8971-4497-aa72-e3b1d47a8125 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:10.761176 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d237db-c463-4575-9f1f-f191e74b28da None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:10.772011 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:10.837269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7d24cb-8971-4497-aa72-e3b1d47a8125 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:10.860246 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cc09bf-3f04-43f1-92f3-d57cb10ccf59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:10.860952 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cc09bf-3f04-43f1-92f3-d57cb10ccf59 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:10.861241 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cc09bf-3f04-43f1-92f3-d57cb10ccf59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:10.861671 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cc09bf-3f04-43f1-92f3-d57cb10ccf59 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:10.862574 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18cc09bf-3f04-43f1-92f3-d57cb10ccf59 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:10.868757 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18cc09bf-3f04-43f1-92f3-d57cb10ccf59 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:10.987181 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 855/1731] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:57:10 2022] POST /identity/v3/projects => generated 409 bytes in 348 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:57:10.988175 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cc09bf-3f04-43f1-92f3-d57cb10ccf59 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:11.012092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a1dc54-72b0-44b0-8f27-e17af174aa42 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:11.012933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a1dc54-72b0-44b0-8f27-e17af174aa42 service nova] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:11.019398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a1dc54-72b0-44b0-8f27-e17af174aa42 service nova] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:11.019881 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a1dc54-72b0-44b0-8f27-e17af174aa42 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:11.186412 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:11.202165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cc09bf-3f04-43f1-92f3-d57cb10ccf59 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:11.220769 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890fefd0-c704-4c51-8131-c51c0ff667b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:11.221460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890fefd0-c704-4c51-8131-c51c0ff667b8 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:11.221851 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890fefd0-c704-4c51-8131-c51c0ff667b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:11.222235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890fefd0-c704-4c51-8131-c51c0ff667b8 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:11.225296 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890fefd0-c704-4c51-8131-c51c0ff667b8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:11.227486 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890fefd0-c704-4c51-8131-c51c0ff667b8 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:11.311218 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 856/1732] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:57:10 2022] POST /identity/v3/users => generated 311 bytes in 321 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:57:11.318679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51ae164-cf78-4032-bc70-b666ea064cec None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:11.319252 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51ae164-cf78-4032-bc70-b666ea064cec None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:11.321419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51ae164-cf78-4032-bc70-b666ea064cec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:11.343228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a1dc54-72b0-44b0-8f27-e17af174aa42 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:11.356624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a1dc54-72b0-44b0-8f27-e17af174aa42 service nova] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:11.405854 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 877/1733] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:10 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 991 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:57:11.408978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a1dc54-72b0-44b0-8f27-e17af174aa42 service nova] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:11.436235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-f51ae164-cf78-4032-bc70-b666ea064cec None None] MFA Rules not processed for user `54d4e437036640b58fda503c4d213f5c`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:11.437806 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-f51ae164-cf78-4032-bc70-b666ea064cec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:11.457436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a1dc54-72b0-44b0-8f27-e17af174aa42 service nova] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:11.473243 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3915f27-f5d6-41b0-bbae-c4aad65cbdc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:11.479144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3915f27-f5d6-41b0-bbae-c4aad65cbdc8 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:11.479559 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3915f27-f5d6-41b0-bbae-c4aad65cbdc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:11.479889 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3915f27-f5d6-41b0-bbae-c4aad65cbdc8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:11.480622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3915f27-f5d6-41b0-bbae-c4aad65cbdc8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:11.482416 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3915f27-f5d6-41b0-bbae-c4aad65cbdc8 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:11.556702 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 878/1734] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:11 2022] GET /identity/v3/roles => generated 1420 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:57:11.560944 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3915f27-f5d6-41b0-bbae-c4aad65cbdc8 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:11.620455 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3915f27-f5d6-41b0-bbae-c4aad65cbdc8 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:11.642542 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b93f5fc-ed14-4cd5-a3bc-264a1853ce66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:11.643238 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b93f5fc-ed14-4cd5-a3bc-264a1853ce66 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:11.643508 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b93f5fc-ed14-4cd5-a3bc-264a1853ce66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:11.643778 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b93f5fc-ed14-4cd5-a3bc-264a1853ce66 admin admin] PATH_INFO: `/v3/projects/73dfec534a5a419b9327c067863a160f/users/c394cf1a20bf49a5b64741d64640cd08/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:11.754071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b93f5fc-ed14-4cd5-a3bc-264a1853ce66 admin admin] RBAC: Authorizing `identity:create_grant(project_id=73dfec534a5a419b9327c067863a160f, user_id=c394cf1a20bf49a5b64741d64640cd08, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:11.777819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b93f5fc-ed14-4cd5-a3bc-264a1853ce66 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:11.784357 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 857/1735] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:11 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 475 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:11.786720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f9cca1-30e9-4142-a30b-fbd54c577d56 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:11.788073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f9cca1-30e9-4142-a30b-fbd54c577d56 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:11.789663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f9cca1-30e9-4142-a30b-fbd54c577d56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:11.813785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 879/1736] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:57:11 2022] PUT /identity/v3/projects/73dfec534a5a419b9327c067863a160f/users/c394cf1a20bf49a5b64741d64640cd08/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:11.820035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b93f5fc-ed14-4cd5-a3bc-264a1853ce66 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:11.963988 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b93f5fc-ed14-4cd5-a3bc-264a1853ce66 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:11.976895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c2a518-2b3d-47a1-aed5-20f0c93ee507 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:11.978420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c2a518-2b3d-47a1-aed5-20f0c93ee507 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:11.979922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c2a518-2b3d-47a1-aed5-20f0c93ee507 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:11.980310 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c2a518-2b3d-47a1-aed5-20f0c93ee507 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:11.982278 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c2a518-2b3d-47a1-aed5-20f0c93ee507 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:11.994439 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c2a518-2b3d-47a1-aed5-20f0c93ee507 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:12.027056 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-a6f9cca1-30e9-4142-a30b-fbd54c577d56 None None] MFA Rules not processed for user `3e31dce69c344c9aa4a55f31b9c735df`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:12.033088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 880/1737] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:11 2022] GET /identity/v3/roles => generated 1420 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:57:12.036451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c2a518-2b3d-47a1-aed5-20f0c93ee507 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:12.038515 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-a6f9cca1-30e9-4142-a30b-fbd54c577d56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:12.083522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c2a518-2b3d-47a1-aed5-20f0c93ee507 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:12.100304 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e647607a-e0f0-4535-a71b-ef1d5a6e8547 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:12.101533 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e647607a-e0f0-4535-a71b-ef1d5a6e8547 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:12.101954 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e647607a-e0f0-4535-a71b-ef1d5a6e8547 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:12.103949 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e647607a-e0f0-4535-a71b-ef1d5a6e8547 admin admin] PATH_INFO: `/v3/projects/73dfec534a5a419b9327c067863a160f/users/c394cf1a20bf49a5b64741d64640cd08/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:12.111341 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e647607a-e0f0-4535-a71b-ef1d5a6e8547 admin admin] RBAC: Authorizing `identity:create_grant(project_id=73dfec534a5a419b9327c067863a160f, user_id=c394cf1a20bf49a5b64741d64640cd08, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:12.132022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e647607a-e0f0-4535-a71b-ef1d5a6e8547 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:12.141666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:12.166059 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 881/1738] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:57:12 2022] PUT /identity/v3/projects/73dfec534a5a419b9327c067863a160f/users/c394cf1a20bf49a5b64741d64640cd08/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:12.169061 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3333a605-fb11-456b-8af3-af1796887913 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:12.169848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3333a605-fb11-456b-8af3-af1796887913 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:12.170619 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3333a605-fb11-456b-8af3-af1796887913 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:12.274942 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-3333a605-fb11-456b-8af3-af1796887913 None None] MFA Rules not processed for user `2f9751adf8d74ab59b6b0a5d00221405`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:12.279779 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3333a605-fb11-456b-8af3-af1796887913 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:12.283763 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 858/1739] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 18 00:57:11 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 498 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:12.288528 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72859a7-9511-4eb9-87c5-c822051b197f None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:12.289634 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72859a7-9511-4eb9-87c5-c822051b197f None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:12.290795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72859a7-9511-4eb9-87c5-c822051b197f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:12.378024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-f72859a7-9511-4eb9-87c5-c822051b197f None None] MFA Rules not processed for user `c394cf1a20bf49a5b64741d64640cd08`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:12.381185 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-f72859a7-9511-4eb9-87c5-c822051b197f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:12.529670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:12.600328 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 859/1740] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:12 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 315 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:12.601218 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72859a7-9511-4eb9-87c5-c822051b197f None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:12.601653 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 882/1741] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:12 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 435 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:12.618398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3333a605-fb11-456b-8af3-af1796887913 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:12.623391 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-f72859a7-9511-4eb9-87c5-c822051b197f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:12.768577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72859a7-9511-4eb9-87c5-c822051b197f None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:12.777261 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92337f44-2aeb-4a56-a84a-b3bb2644c0ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:12.778037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92337f44-2aeb-4a56-a84a-b3bb2644c0ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:12.778420 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92337f44-2aeb-4a56-a84a-b3bb2644c0ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:12.778791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92337f44-2aeb-4a56-a84a-b3bb2644c0ab admin admin] PATH_INFO: `/v3/users/1f80c6fb434f46238c4999197703f98d` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:12.780666 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92337f44-2aeb-4a56-a84a-b3bb2644c0ab admin admin] RBAC: Authorizing `identity:delete_user(user_id=1f80c6fb434f46238c4999197703f98d)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:12.784775 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92337f44-2aeb-4a56-a84a-b3bb2644c0ab admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:12.851395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3333a605-fb11-456b-8af3-af1796887913 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:12.869333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad7bd8f-a1c1-43c2-a412-bcaaf54f46a3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:12.870292 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad7bd8f-a1c1-43c2-a412-bcaaf54f46a3 service nova] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:12.872029 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad7bd8f-a1c1-43c2-a412-bcaaf54f46a3 service nova] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:12.872479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad7bd8f-a1c1-43c2-a412-bcaaf54f46a3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:12.895101 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-92337f44-2aeb-4a56-a84a-b3bb2644c0ab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1f80c6fb434f46238c4999197703f98d'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:12.904594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad7bd8f-a1c1-43c2-a412-bcaaf54f46a3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:12.909343 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad7bd8f-a1c1-43c2-a412-bcaaf54f46a3 service nova] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:12.915326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-92337f44-2aeb-4a56-a84a-b3bb2644c0ab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1f80c6fb434f46238c4999197703f98d'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:12.916389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-92337f44-2aeb-4a56-a84a-b3bb2644c0ab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1f80c6fb434f46238c4999197703f98d'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:12.917184 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-92337f44-2aeb-4a56-a84a-b3bb2644c0ab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1f80c6fb434f46238c4999197703f98d'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:12.941566 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-92337f44-2aeb-4a56-a84a-b3bb2644c0ab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1f80c6fb434f46238c4999197703f98d'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:12.974376 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 883/1742] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:12 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:12.981019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad7bd8f-a1c1-43c2-a412-bcaaf54f46a3 service nova] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:12.989513 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-aad7bd8f-a1c1-43c2-a412-bcaaf54f46a3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:13.016454 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 860/1743] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:57:12 2022] DELETE /identity/v3/users/1f80c6fb434f46238c4999197703f98d => generated 0 bytes in 415 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:13.021250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92337f44-2aeb-4a56-a84a-b3bb2644c0ab admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:13.027860 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-92337f44-2aeb-4a56-a84a-b3bb2644c0ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:13.156987 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad7bd8f-a1c1-43c2-a412-bcaaf54f46a3 service nova] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:13.199974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:13.257664 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:13.289401 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fca706-9a59-4bcc-b180-d0fecb148f30 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:13.290361 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fca706-9a59-4bcc-b180-d0fecb148f30 service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:13.290765 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fca706-9a59-4bcc-b180-d0fecb148f30 service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:13.291043 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fca706-9a59-4bcc-b180-d0fecb148f30 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:13.297080 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-87fca706-9a59-4bcc-b180-d0fecb148f30 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:13.319940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92337f44-2aeb-4a56-a84a-b3bb2644c0ab admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:13.342908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0398b6b-34b6-484b-9af2-b5c87a4535ce service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6230ed7ce0eb43e9b2b8530a28eb6299', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:13.343973 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0398b6b-34b6-484b-9af2-b5c87a4535ce service swift] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:13.349105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0398b6b-34b6-484b-9af2-b5c87a4535ce service swift] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:13.350090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0398b6b-34b6-484b-9af2-b5c87a4535ce service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:13.358050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-a0398b6b-34b6-484b-9af2-b5c87a4535ce service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:13.542877 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fca706-9a59-4bcc-b180-d0fecb148f30 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:13.544350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fca706-9a59-4bcc-b180-d0fecb148f30 service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:13.577817 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0398b6b-34b6-484b-9af2-b5c87a4535ce service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:13.579848 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0398b6b-34b6-484b-9af2-b5c87a4535ce service swift] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:13.596239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 884/1744] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:12 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 621 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:13.597852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1fe5d5-e5d7-4091-a47f-e2046fdf08d5 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:13.598292 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1fe5d5-e5d7-4091-a47f-e2046fdf08d5 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:13.600207 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1fe5d5-e5d7-4091-a47f-e2046fdf08d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:13.603414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 861/1745] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 18 00:57:13 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 587 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 18 00:57:13.688077 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-9b1fe5d5-e5d7-4091-a47f-e2046fdf08d5 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:13.692713 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-9b1fe5d5-e5d7-4091-a47f-e2046fdf08d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:14.048825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 885/1746] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 452 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:14.114146 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0398b6b-34b6-484b-9af2-b5c87a4535ce service swift] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:14.116222 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-a0398b6b-34b6-484b-9af2-b5c87a4535ce service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:14.327533 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0398b6b-34b6-484b-9af2-b5c87a4535ce service swift] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:14.335912 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1fe5d5-e5d7-4091-a47f-e2046fdf08d5 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:14.337803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-9b1fe5d5-e5d7-4091-a47f-e2046fdf08d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:14.342027 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b08b2d-a49c-4184-b4b4-88390772a0a2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:14.342927 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b08b2d-a49c-4184-b4b4-88390772a0a2 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:14.347857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b08b2d-a49c-4184-b4b4-88390772a0a2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:14.348509 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b08b2d-a49c-4184-b4b4-88390772a0a2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:14.364182 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1fe5d5-e5d7-4091-a47f-e2046fdf08d5 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:14.379800 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064d45b6-5681-47f1-893f-81dbb8373984 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:14.380617 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b08b2d-a49c-4184-b4b4-88390772a0a2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:14.382574 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b08b2d-a49c-4184-b4b4-88390772a0a2 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:14.387114 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064d45b6-5681-47f1-893f-81dbb8373984 service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:14.388472 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064d45b6-5681-47f1-893f-81dbb8373984 service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:14.389120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064d45b6-5681-47f1-893f-81dbb8373984 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:14.398790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-064d45b6-5681-47f1-893f-81dbb8373984 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:14.418839 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 862/1747] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:57:14.627305 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064d45b6-5681-47f1-893f-81dbb8373984 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:14.646038 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064d45b6-5681-47f1-893f-81dbb8373984 service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:14.705541 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 886/1748] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:14 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:17.468040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa4c231-0ae9-4d96-a5c6-b739ab110846 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:17.473072 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa4c231-0ae9-4d96-a5c6-b739ab110846 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:17.473943 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa4c231-0ae9-4d96-a5c6-b739ab110846 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:17.566907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-8aa4c231-0ae9-4d96-a5c6-b739ab110846 None None] MFA Rules not processed for user `9b0510812b60454586a781e6399e77a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:17.568795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8aa4c231-0ae9-4d96-a5c6-b739ab110846 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:17.597038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 863/1749] 10.222.0.49 () {58 vars in 1015 bytes} [Sun Sep 18 00:57:17 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:57:17.629160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064d45b6-5681-47f1-893f-81dbb8373984 service cinder] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:17.648974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064d45b6-5681-47f1-893f-81dbb8373984 service cinder] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:17.673584 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e2a636-e992-4a24-a02f-1eac5ae640c0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:17.675186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e2a636-e992-4a24-a02f-1eac5ae640c0 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:17.676613 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e2a636-e992-4a24-a02f-1eac5ae640c0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:17.679983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e2a636-e992-4a24-a02f-1eac5ae640c0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:17.711855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e2a636-e992-4a24-a02f-1eac5ae640c0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:17.720089 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e2a636-e992-4a24-a02f-1eac5ae640c0 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:17.735541 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:17.758829 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 887/1750] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:17 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:18.861614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa4c231-0ae9-4d96-a5c6-b739ab110846 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:18.863750 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8aa4c231-0ae9-4d96-a5c6-b739ab110846 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:18.902816 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa4c231-0ae9-4d96-a5c6-b739ab110846 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:18.921238 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d28d82d-51e2-4b7c-af69-f684f9191f2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:18.921918 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d28d82d-51e2-4b7c-af69-f684f9191f2e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:18.922195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d28d82d-51e2-4b7c-af69-f684f9191f2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:18.922482 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d28d82d-51e2-4b7c-af69-f684f9191f2e admin admin] PATH_INFO: `/v3/projects/5f2951e9bd2844deb85847ffd121ae3a` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:18.930242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d28d82d-51e2-4b7c-af69-f684f9191f2e admin admin] RBAC: Authorizing `identity:delete_project(project_id=5f2951e9bd2844deb85847ffd121ae3a)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:18.932336 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d28d82d-51e2-4b7c-af69-f684f9191f2e admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:19.045657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-1d28d82d-51e2-4b7c-af69-f684f9191f2e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5f2951e9bd2844deb85847ffd121ae3a', 'request_id': 'req-1d28d82d-51e2-4b7c-af69-f684f9191f2e'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:19.057251 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-1d28d82d-51e2-4b7c-af69-f684f9191f2e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5f2951e9bd2844deb85847ffd121ae3a', 'request_id': 'req-1d28d82d-51e2-4b7c-af69-f684f9191f2e'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:19.086744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-1d28d82d-51e2-4b7c-af69-f684f9191f2e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5f2951e9bd2844deb85847ffd121ae3a', 'request_id': 'req-1d28d82d-51e2-4b7c-af69-f684f9191f2e'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:19.136081 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-1d28d82d-51e2-4b7c-af69-f684f9191f2e admin admin] The token cache is being invalidate because project 5f2951e9bd2844deb85847ffd121ae3a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:57:19.136524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-1d28d82d-51e2-4b7c-af69-f684f9191f2e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:19.142518 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-1d28d82d-51e2-4b7c-af69-f684f9191f2e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:19.144398 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 864/1751] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:57:18 2022] DELETE /identity/v3/projects/5f2951e9bd2844deb85847ffd121ae3a => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:19.149746 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e2a636-e992-4a24-a02f-1eac5ae640c0 service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:19.171749 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e2a636-e992-4a24-a02f-1eac5ae640c0 service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:19.324044 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68720a58-204e-4bb5-add2-3d4fdfd4c899 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:19.324713 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68720a58-204e-4bb5-add2-3d4fdfd4c899 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:19.324996 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68720a58-204e-4bb5-add2-3d4fdfd4c899 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:19.325273 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68720a58-204e-4bb5-add2-3d4fdfd4c899 admin admin] PATH_INFO: `/v3/users/0fd093470e0b49d69966e477691cb926` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:19.327056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68720a58-204e-4bb5-add2-3d4fdfd4c899 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0fd093470e0b49d69966e477691cb926)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:19.329353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68720a58-204e-4bb5-add2-3d4fdfd4c899 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:19.440016 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-68720a58-204e-4bb5-add2-3d4fdfd4c899 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0fd093470e0b49d69966e477691cb926'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:19.457647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-68720a58-204e-4bb5-add2-3d4fdfd4c899 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0fd093470e0b49d69966e477691cb926'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:19.484274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-68720a58-204e-4bb5-add2-3d4fdfd4c899 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0fd093470e0b49d69966e477691cb926'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:19.492021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-68720a58-204e-4bb5-add2-3d4fdfd4c899 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0fd093470e0b49d69966e477691cb926'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:19.508930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-68720a58-204e-4bb5-add2-3d4fdfd4c899 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0fd093470e0b49d69966e477691cb926'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:19.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 888/1752] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:57:19 2022] DELETE /identity/v3/users/0fd093470e0b49d69966e477691cb926 => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:19.635466 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11fad9b-f2b4-4e2f-babe-8f0ae09bca38 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:19.636006 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11fad9b-f2b4-4e2f-babe-8f0ae09bca38 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:19.636460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11fad9b-f2b4-4e2f-babe-8f0ae09bca38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:19.683286 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-a11fad9b-f2b4-4e2f-babe-8f0ae09bca38 None None] MFA Rules not processed for user `294ae98a8492468893beb3b8c2870d5e`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:19.685134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-a11fad9b-f2b4-4e2f-babe-8f0ae09bca38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:19.694288 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:19.820512 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 865/1753] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:19 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:20.039989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68720a58-204e-4bb5-add2-3d4fdfd4c899 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:20.041815 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-68720a58-204e-4bb5-add2-3d4fdfd4c899 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:20.153394 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68720a58-204e-4bb5-add2-3d4fdfd4c899 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:20.158512 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e434772-ad58-4eb5-afd4-901b2976881b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:20.159829 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e434772-ad58-4eb5-afd4-901b2976881b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:20.160111 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e434772-ad58-4eb5-afd4-901b2976881b admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:20.160588 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e434772-ad58-4eb5-afd4-901b2976881b admin admin] PATH_INFO: `/v3/projects/d746b9566bbc451c935207207ca4da5c` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:20.161789 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e434772-ad58-4eb5-afd4-901b2976881b admin admin] RBAC: Authorizing `identity:delete_project(project_id=d746b9566bbc451c935207207ca4da5c)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:20.164102 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e434772-ad58-4eb5-afd4-901b2976881b admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:20.201540 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:20.246170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7e434772-ad58-4eb5-afd4-901b2976881b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd746b9566bbc451c935207207ca4da5c', 'request_id': 'req-7e434772-ad58-4eb5-afd4-901b2976881b'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:20.255648 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7e434772-ad58-4eb5-afd4-901b2976881b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd746b9566bbc451c935207207ca4da5c', 'request_id': 'req-7e434772-ad58-4eb5-afd4-901b2976881b'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:20.277162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7e434772-ad58-4eb5-afd4-901b2976881b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd746b9566bbc451c935207207ca4da5c', 'request_id': 'req-7e434772-ad58-4eb5-afd4-901b2976881b'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:20.296974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7e434772-ad58-4eb5-afd4-901b2976881b admin admin] The token cache is being invalidate because project d746b9566bbc451c935207207ca4da5c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:57:20.296974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7e434772-ad58-4eb5-afd4-901b2976881b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:20.297677 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7e434772-ad58-4eb5-afd4-901b2976881b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:20.299664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 889/1754] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:57:20 2022] DELETE /identity/v3/projects/d746b9566bbc451c935207207ca4da5c => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:20.303448 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11fad9b-f2b4-4e2f-babe-8f0ae09bca38 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:20.320530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11fad9b-f2b4-4e2f-babe-8f0ae09bca38 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:20.430340 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1bf516c-db18-40a2-8119-dae3bdb185c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:20.431048 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bf516c-db18-40a2-8119-dae3bdb185c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:20.431356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bf516c-db18-40a2-8119-dae3bdb185c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:20.431604 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bf516c-db18-40a2-8119-dae3bdb185c7 admin admin] PATH_INFO: `/v3/users/b2801b77031544e7ad0bcafca672bfd8` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:20.432835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1bf516c-db18-40a2-8119-dae3bdb185c7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b2801b77031544e7ad0bcafca672bfd8)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:20.435290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1bf516c-db18-40a2-8119-dae3bdb185c7 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:20.526972 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-a1bf516c-db18-40a2-8119-dae3bdb185c7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b2801b77031544e7ad0bcafca672bfd8'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:20.554088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-a1bf516c-db18-40a2-8119-dae3bdb185c7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b2801b77031544e7ad0bcafca672bfd8'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:20.554803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-a1bf516c-db18-40a2-8119-dae3bdb185c7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b2801b77031544e7ad0bcafca672bfd8'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:20.555476 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-a1bf516c-db18-40a2-8119-dae3bdb185c7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b2801b77031544e7ad0bcafca672bfd8'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:20.578202 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-a1bf516c-db18-40a2-8119-dae3bdb185c7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b2801b77031544e7ad0bcafca672bfd8'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:20.610223 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 866/1755] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:57:20 2022] DELETE /identity/v3/users/b2801b77031544e7ad0bcafca672bfd8 => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:57:21.429252 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e434772-ad58-4eb5-afd4-901b2976881b admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:21.431465 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-7e434772-ad58-4eb5-afd4-901b2976881b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:21.483999 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e434772-ad58-4eb5-afd4-901b2976881b admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:21.594359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10146418-cb4e-4475-837e-8e62ceb09b29 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:21.594879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10146418-cb4e-4475-837e-8e62ceb09b29 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:21.594972 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10146418-cb4e-4475-837e-8e62ceb09b29 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:21.598032 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 867/1756] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 18 00:57:21 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:57:21.660442 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0eca8c-9144-4325-ba6c-73155421b20a None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:21.660725 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0eca8c-9144-4325-ba6c-73155421b20a None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:21.661377 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0eca8c-9144-4325-ba6c-73155421b20a None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:21.663371 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 868/1757] 10.222.0.42 () {54 vars in 912 bytes} [Sun Sep 18 00:57:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 18 00:57:21.678303 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c6c658-40a7-4ef0-87fc-255f11d03c35 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:21.678873 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c6c658-40a7-4ef0-87fc-255f11d03c35 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:21.679336 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c6c658-40a7-4ef0-87fc-255f11d03c35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:21.701175 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a4cad1-f9ee-4e25-8d5f-731ca28e6420 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:21.701956 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a4cad1-f9ee-4e25-8d5f-731ca28e6420 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:21.702240 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a4cad1-f9ee-4e25-8d5f-731ca28e6420 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:21.702538 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a4cad1-f9ee-4e25-8d5f-731ca28e6420 admin admin] PATH_INFO: `/v3/projects/378035cecce241af8f09636c8f412eb8` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:21.707331 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a4cad1-f9ee-4e25-8d5f-731ca28e6420 admin admin] RBAC: Authorizing `identity:delete_project(project_id=378035cecce241af8f09636c8f412eb8)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:21.714818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a4cad1-f9ee-4e25-8d5f-731ca28e6420 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:21.804121 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-24c6c658-40a7-4ef0-87fc-255f11d03c35 None None] MFA Rules not processed for user `9b0510812b60454586a781e6399e77a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:21.812686 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-24c6c658-40a7-4ef0-87fc-255f11d03c35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:21.897093 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-66a4cad1-f9ee-4e25-8d5f-731ca28e6420 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '378035cecce241af8f09636c8f412eb8', 'request_id': 'req-66a4cad1-f9ee-4e25-8d5f-731ca28e6420'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:21.922184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-66a4cad1-f9ee-4e25-8d5f-731ca28e6420 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '378035cecce241af8f09636c8f412eb8', 'request_id': 'req-66a4cad1-f9ee-4e25-8d5f-731ca28e6420'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:21.926653 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:21.947354 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-66a4cad1-f9ee-4e25-8d5f-731ca28e6420 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '378035cecce241af8f09636c8f412eb8', 'request_id': 'req-66a4cad1-f9ee-4e25-8d5f-731ca28e6420'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:21.978813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-66a4cad1-f9ee-4e25-8d5f-731ca28e6420 admin admin] The token cache is being invalidate because project 378035cecce241af8f09636c8f412eb8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:57:21.978813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-66a4cad1-f9ee-4e25-8d5f-731ca28e6420 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:21.980044 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-66a4cad1-f9ee-4e25-8d5f-731ca28e6420 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:21.982112 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 890/1758] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:57:21 2022] DELETE /identity/v3/projects/378035cecce241af8f09636c8f412eb8 => generated 0 bytes in 553 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:21.983014 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b71c58-af44-42aa-95af-d213872faefc None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:21.983258 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b71c58-af44-42aa-95af-d213872faefc None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:21.983537 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b71c58-af44-42aa-95af-d213872faefc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:22.009499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 869/1759] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 18 00:57:21 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 333 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:57:22.014974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f14367-8022-4a41-a4de-beb383a0f450 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:22.016595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f14367-8022-4a41-a4de-beb383a0f450 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:22.016595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f14367-8022-4a41-a4de-beb383a0f450 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:22.035995 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-c6b71c58-af44-42aa-95af-d213872faefc None None] MFA Rules not processed for user `a59d73fd45a54651930d52300704fdb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:22.037821 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c6b71c58-af44-42aa-95af-d213872faefc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:22.105042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-49f14367-8022-4a41-a4de-beb383a0f450 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:22.106349 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-49f14367-8022-4a41-a4de-beb383a0f450 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:22.161050 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:22.306771 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 891/1760] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 18 00:57:21 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 324 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:22.308070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b71c58-af44-42aa-95af-d213872faefc None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:22.310487 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c6b71c58-af44-42aa-95af-d213872faefc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:22.364621 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 870/1761] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 354 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:22.376986 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3760352f-ce92-4dda-b655-57ef7d48553e None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:22.377309 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3760352f-ce92-4dda-b655-57ef7d48553e None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:22.377614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3760352f-ce92-4dda-b655-57ef7d48553e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:22.414600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-3760352f-ce92-4dda-b655-57ef7d48553e None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:22.415831 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-3760352f-ce92-4dda-b655-57ef7d48553e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:22.454950 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 871/1762] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:22.455956 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3760352f-ce92-4dda-b655-57ef7d48553e None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:22.491958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3760352f-ce92-4dda-b655-57ef7d48553e None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:22.507313 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eae8f7c-d354-4d84-b71e-4271626c08a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:22.507953 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eae8f7c-d354-4d84-b71e-4271626c08a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:22.508238 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eae8f7c-d354-4d84-b71e-4271626c08a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:22.508511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eae8f7c-d354-4d84-b71e-4271626c08a2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:22.518275 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eae8f7c-d354-4d84-b71e-4271626c08a2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:57:22.525848 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eae8f7c-d354-4d84-b71e-4271626c08a2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:22.539409 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eae8f7c-d354-4d84-b71e-4271626c08a2 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:22.571684 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 872/1763] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:57:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:57:22.572843 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eae8f7c-d354-4d84-b71e-4271626c08a2 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:22.591158 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eae8f7c-d354-4d84-b71e-4271626c08a2 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:22.594268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b71c58-af44-42aa-95af-d213872faefc None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:22.605007 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a491621d-e26a-4800-af44-22ce705d649a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:22.605645 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a491621d-e26a-4800-af44-22ce705d649a service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:22.605929 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a491621d-e26a-4800-af44-22ce705d649a service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:22.606206 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a491621d-e26a-4800-af44-22ce705d649a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:22.608269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeaad13a-ecda-4565-8a62-8d87b5b34f7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:22.608904 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeaad13a-ecda-4565-8a62-8d87b5b34f7f admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:22.609186 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeaad13a-ecda-4565-8a62-8d87b5b34f7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:22.609463 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeaad13a-ecda-4565-8a62-8d87b5b34f7f admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:22.610120 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeaad13a-ecda-4565-8a62-8d87b5b34f7f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:22.614174 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeaad13a-ecda-4565-8a62-8d87b5b34f7f admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:22.644938 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a491621d-e26a-4800-af44-22ce705d649a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:22.654305 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a491621d-e26a-4800-af44-22ce705d649a service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:22.662267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 873/1764] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:57:22 2022] POST /identity/v3/projects => generated 373 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:57:22.665982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeaad13a-ecda-4565-8a62-8d87b5b34f7f admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:22.694531 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 892/1765] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:22 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:22.810411 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeaad13a-ecda-4565-8a62-8d87b5b34f7f admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:22.826739 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4f1eef-2a01-456b-a997-5103600d5f32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:22.827379 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4f1eef-2a01-456b-a997-5103600d5f32 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:22.827663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4f1eef-2a01-456b-a997-5103600d5f32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:22.827935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4f1eef-2a01-456b-a997-5103600d5f32 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:22.828935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c4f1eef-2a01-456b-a997-5103600d5f32 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:22.832481 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c4f1eef-2a01-456b-a997-5103600d5f32 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:22.934491 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 874/1766] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:57:22 2022] POST /identity/v3/users => generated 287 bytes in 267 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:57:22.939306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a491621d-e26a-4800-af44-22ce705d649a service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:22.959334 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a491621d-e26a-4800-af44-22ce705d649a service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:23.019611 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73d8374-8019-4988-9a44-feb9715096e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:23.028052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73d8374-8019-4988-9a44-feb9715096e4 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:23.028450 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73d8374-8019-4988-9a44-feb9715096e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:23.034931 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73d8374-8019-4988-9a44-feb9715096e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:23.039381 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73d8374-8019-4988-9a44-feb9715096e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:23.048128 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73d8374-8019-4988-9a44-feb9715096e4 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:23.088536 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 893/1767] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:22 2022] GET /identity/v3/roles => generated 1420 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:57:23.092107 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4f1eef-2a01-456b-a997-5103600d5f32 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:23.113617 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4f1eef-2a01-456b-a997-5103600d5f32 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:23.128393 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd51e5ee-64ee-4232-aa2d-9947605f3bba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:23.131276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd51e5ee-64ee-4232-aa2d-9947605f3bba admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:23.131773 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd51e5ee-64ee-4232-aa2d-9947605f3bba admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:23.132211 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd51e5ee-64ee-4232-aa2d-9947605f3bba admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:23.133061 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd51e5ee-64ee-4232-aa2d-9947605f3bba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:23.139517 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd51e5ee-64ee-4232-aa2d-9947605f3bba admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:23.173431 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 875/1768] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:23 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:57:23.176111 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73d8374-8019-4988-9a44-feb9715096e4 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:23.199622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73d8374-8019-4988-9a44-feb9715096e4 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:23.214554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17e20c7-fcdf-41c8-bd79-59ba94c4db04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:23.216639 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17e20c7-fcdf-41c8-bd79-59ba94c4db04 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:23.217086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17e20c7-fcdf-41c8-bd79-59ba94c4db04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:23.217487 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17e20c7-fcdf-41c8-bd79-59ba94c4db04 admin admin] PATH_INFO: `/v3/projects/2add46ff01ea465fb8275c01739ef48e/users/41b029d4a5be4a6ab4ce329731fe1ade/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:23.262730 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17e20c7-fcdf-41c8-bd79-59ba94c4db04 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2add46ff01ea465fb8275c01739ef48e, user_id=41b029d4a5be4a6ab4ce329731fe1ade, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:23.268771 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17e20c7-fcdf-41c8-bd79-59ba94c4db04 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:23.320405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 894/1769] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:57:23 2022] PUT /identity/v3/projects/2add46ff01ea465fb8275c01739ef48e/users/41b029d4a5be4a6ab4ce329731fe1ade/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:23.328144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5cb838-e318-40eb-a834-89a120bc0564 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:23.328617 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5cb838-e318-40eb-a834-89a120bc0564 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:23.328825 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5cb838-e318-40eb-a834-89a120bc0564 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:23.451907 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17e20c7-fcdf-41c8-bd79-59ba94c4db04 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:23.456152 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c17e20c7-fcdf-41c8-bd79-59ba94c4db04 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:23.457476 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-8d5cb838-e318-40eb-a834-89a120bc0564 None None] MFA Rules not processed for user `41b029d4a5be4a6ab4ce329731fe1ade`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:23.470336 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8d5cb838-e318-40eb-a834-89a120bc0564 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:23.501651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:23.787505 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 876/1770] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:23 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 465 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:23.794491 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe0d9b3-9548-4cc5-bfbc-7c72fe07dab4 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:23.795787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe0d9b3-9548-4cc5-bfbc-7c72fe07dab4 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:23.795787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe0d9b3-9548-4cc5-bfbc-7c72fe07dab4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:23.819995 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17e20c7-fcdf-41c8-bd79-59ba94c4db04 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:23.843734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a172264d-ed8f-42f0-b4e9-7c25edb14037 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:23.846203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a172264d-ed8f-42f0-b4e9-7c25edb14037 service nova] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:23.846203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a172264d-ed8f-42f0-b4e9-7c25edb14037 service nova] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:23.846203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a172264d-ed8f-42f0-b4e9-7c25edb14037 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:23.859200 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-bbe0d9b3-9548-4cc5-bfbc-7c72fe07dab4 None None] MFA Rules not processed for user `2f9751adf8d74ab59b6b0a5d00221405`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:23.862357 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-bbe0d9b3-9548-4cc5-bfbc-7c72fe07dab4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:23.895904 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a172264d-ed8f-42f0-b4e9-7c25edb14037 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:23.901549 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a172264d-ed8f-42f0-b4e9-7c25edb14037 service nova] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:23.956964 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 895/1771] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:23 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 502 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:23.958298 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a172264d-ed8f-42f0-b4e9-7c25edb14037 service nova] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:23.963455 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a172264d-ed8f-42f0-b4e9-7c25edb14037 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:24.037652 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:24.061467 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 877/1772] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:23 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 273 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:24.132778 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a172264d-ed8f-42f0-b4e9-7c25edb14037 service nova] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:24.141736 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0bea70-6776-4a13-ab96-6bb8cfb5f167 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:24.142692 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0bea70-6776-4a13-ab96-6bb8cfb5f167 service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:24.143087 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0bea70-6776-4a13-ab96-6bb8cfb5f167 service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:24.143530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0bea70-6776-4a13-ab96-6bb8cfb5f167 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:24.166977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0bea70-6776-4a13-ab96-6bb8cfb5f167 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:24.168863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0bea70-6776-4a13-ab96-6bb8cfb5f167 service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:24.208212 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 896/1773] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:23 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 250 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:24.233680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe0d9b3-9548-4cc5-bfbc-7c72fe07dab4 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:24.538315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe0d9b3-9548-4cc5-bfbc-7c72fe07dab4 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:24.559883 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45171219-5774-48bb-b5ab-23235e5386a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:24.560772 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45171219-5774-48bb-b5ab-23235e5386a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:24.561007 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45171219-5774-48bb-b5ab-23235e5386a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:24.561291 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45171219-5774-48bb-b5ab-23235e5386a3 admin admin] PATH_INFO: `/v3/users/41b029d4a5be4a6ab4ce329731fe1ade` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:24.567706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45171219-5774-48bb-b5ab-23235e5386a3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=41b029d4a5be4a6ab4ce329731fe1ade)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:24.569737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45171219-5774-48bb-b5ab-23235e5386a3 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:24.710738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-45171219-5774-48bb-b5ab-23235e5386a3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '41b029d4a5be4a6ab4ce329731fe1ade'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:24.762575 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-45171219-5774-48bb-b5ab-23235e5386a3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '41b029d4a5be4a6ab4ce329731fe1ade'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:24.782661 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-45171219-5774-48bb-b5ab-23235e5386a3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '41b029d4a5be4a6ab4ce329731fe1ade'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:24.784839 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-45171219-5774-48bb-b5ab-23235e5386a3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '41b029d4a5be4a6ab4ce329731fe1ade'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:24.805714 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:24.839569 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-45171219-5774-48bb-b5ab-23235e5386a3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '41b029d4a5be4a6ab4ce329731fe1ade'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:24.978027 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 878/1774] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:57:24 2022] DELETE /identity/v3/users/41b029d4a5be4a6ab4ce329731fe1ade => generated 0 bytes in 741 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:24.979936 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91314d3-cf75-4f12-977a-ae428d1288c7 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:24.980424 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91314d3-cf75-4f12-977a-ae428d1288c7 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:24.980504 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91314d3-cf75-4f12-977a-ae428d1288c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:25.064768 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-b91314d3-cf75-4f12-977a-ae428d1288c7 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:25.076217 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-b91314d3-cf75-4f12-977a-ae428d1288c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:25.209933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 897/1775] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:25.265889 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45171219-5774-48bb-b5ab-23235e5386a3 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:25.270424 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-45171219-5774-48bb-b5ab-23235e5386a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:25.487036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45171219-5774-48bb-b5ab-23235e5386a3 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:25.498083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e7a4bb-c224-47a4-ba92-a8f3959c15fe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:25.499165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e7a4bb-c224-47a4-ba92-a8f3959c15fe service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:25.506089 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e7a4bb-c224-47a4-ba92-a8f3959c15fe service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:25.506089 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e7a4bb-c224-47a4-ba92-a8f3959c15fe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:25.534788 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e7a4bb-c224-47a4-ba92-a8f3959c15fe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:25.544469 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e7a4bb-c224-47a4-ba92-a8f3959c15fe service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:25.601110 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 879/1776] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:26.056075 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd670de-936b-4704-ad40-2eb5effe4a06 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:26.056651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd670de-936b-4704-ad40-2eb5effe4a06 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:26.056651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd670de-936b-4704-ad40-2eb5effe4a06 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:26.059652 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 898/1777] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 18 00:57:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:57:26.279170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f4a26c-bd57-40f2-b4fb-8ec58889d4ce None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:26.279734 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f4a26c-bd57-40f2-b4fb-8ec58889d4ce None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:26.279966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f4a26c-bd57-40f2-b4fb-8ec58889d4ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:26.321241 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-d4f4a26c-bd57-40f2-b4fb-8ec58889d4ce None None] MFA Rules not processed for user `a59d73fd45a54651930d52300704fdb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:26.322743 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d4f4a26c-bd57-40f2-b4fb-8ec58889d4ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:26.500643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 880/1778] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 18 00:57:26 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:26.533209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd670de-936b-4704-ad40-2eb5effe4a06 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:26.537937 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-edd670de-936b-4704-ad40-2eb5effe4a06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:26.591047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd670de-936b-4704-ad40-2eb5effe4a06 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:26.632533 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb1a4a0-f56b-4329-890b-b917854ab2be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:26.633483 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb1a4a0-f56b-4329-890b-b917854ab2be admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:26.633926 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb1a4a0-f56b-4329-890b-b917854ab2be admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:26.636106 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb1a4a0-f56b-4329-890b-b917854ab2be admin admin] PATH_INFO: `/v3/users/1a5a1af071224f4988b2f244519a8396` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:26.640075 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb1a4a0-f56b-4329-890b-b917854ab2be admin admin] RBAC: Authorizing `identity:delete_user(user_id=1a5a1af071224f4988b2f244519a8396)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:26.645723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb1a4a0-f56b-4329-890b-b917854ab2be admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:26.760768 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-adb1a4a0-f56b-4329-890b-b917854ab2be admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1a5a1af071224f4988b2f244519a8396'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:26.786196 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-adb1a4a0-f56b-4329-890b-b917854ab2be admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1a5a1af071224f4988b2f244519a8396'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:26.893154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-adb1a4a0-f56b-4329-890b-b917854ab2be admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1a5a1af071224f4988b2f244519a8396'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:26.894902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-adb1a4a0-f56b-4329-890b-b917854ab2be admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1a5a1af071224f4988b2f244519a8396'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:26.925845 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-adb1a4a0-f56b-4329-890b-b917854ab2be admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1a5a1af071224f4988b2f244519a8396'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:26.957772 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 899/1779] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:57:26 2022] DELETE /identity/v3/users/1a5a1af071224f4988b2f244519a8396 => generated 0 bytes in 425 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:26.963037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b97fd4f-7b95-4f8e-b1e1-a9ff5a440be9 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:26.963380 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b97fd4f-7b95-4f8e-b1e1-a9ff5a440be9 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:26.963664 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b97fd4f-7b95-4f8e-b1e1-a9ff5a440be9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:27.034996 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-7b97fd4f-7b95-4f8e-b1e1-a9ff5a440be9 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:27.041393 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-7b97fd4f-7b95-4f8e-b1e1-a9ff5a440be9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:27.197029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 881/1780] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:27.311387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb1a4a0-f56b-4329-890b-b917854ab2be admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:27.313720 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-adb1a4a0-f56b-4329-890b-b917854ab2be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:27.468756 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:27.509355 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb1a4a0-f56b-4329-890b-b917854ab2be admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:27.525528 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd903b0-30ec-4882-bb98-61af05885127 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:27.526683 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd903b0-30ec-4882-bb98-61af05885127 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:27.527402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd903b0-30ec-4882-bb98-61af05885127 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:27.527890 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd903b0-30ec-4882-bb98-61af05885127 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:27.546307 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd903b0-30ec-4882-bb98-61af05885127 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:27.548791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd903b0-30ec-4882-bb98-61af05885127 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:27.587393 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b97fd4f-7b95-4f8e-b1e1-a9ff5a440be9 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:27.588802 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-7b97fd4f-7b95-4f8e-b1e1-a9ff5a440be9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:27.618286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 900/1781] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:27.633936 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:27.825179 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd903b0-30ec-4882-bb98-61af05885127 service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:27.827213 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-7fd903b0-30ec-4882-bb98-61af05885127 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:27.878419 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd903b0-30ec-4882-bb98-61af05885127 service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:27.903327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43546d3c-88e7-409a-a7e0-b8f7babc5aef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:27.903985 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43546d3c-88e7-409a-a7e0-b8f7babc5aef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:27.904264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43546d3c-88e7-409a-a7e0-b8f7babc5aef admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:27.904538 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43546d3c-88e7-409a-a7e0-b8f7babc5aef admin admin] PATH_INFO: `/v3/projects/2add46ff01ea465fb8275c01739ef48e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:27.909772 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43546d3c-88e7-409a-a7e0-b8f7babc5aef admin admin] RBAC: Authorizing `identity:delete_project(project_id=2add46ff01ea465fb8275c01739ef48e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:27.912538 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43546d3c-88e7-409a-a7e0-b8f7babc5aef admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:27.955621 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b97fd4f-7b95-4f8e-b1e1-a9ff5a440be9 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:27.976164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329d3bdb-03b5-442a-86e2-5d691ba2e1ad service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:27.976801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329d3bdb-03b5-442a-86e2-5d691ba2e1ad service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:27.977079 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329d3bdb-03b5-442a-86e2-5d691ba2e1ad service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:27.977349 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329d3bdb-03b5-442a-86e2-5d691ba2e1ad service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:27.981287 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-329d3bdb-03b5-442a-86e2-5d691ba2e1ad service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:28.010948 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329d3bdb-03b5-442a-86e2-5d691ba2e1ad service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:28.012776 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329d3bdb-03b5-442a-86e2-5d691ba2e1ad service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:28.027155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-43546d3c-88e7-409a-a7e0-b8f7babc5aef admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2add46ff01ea465fb8275c01739ef48e', 'request_id': 'req-43546d3c-88e7-409a-a7e0-b8f7babc5aef'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:28.037782 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 882/1782] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:27 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 450 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:28.038954 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-43546d3c-88e7-409a-a7e0-b8f7babc5aef admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2add46ff01ea465fb8275c01739ef48e', 'request_id': 'req-43546d3c-88e7-409a-a7e0-b8f7babc5aef'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:28.047940 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-43546d3c-88e7-409a-a7e0-b8f7babc5aef admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2add46ff01ea465fb8275c01739ef48e', 'request_id': 'req-43546d3c-88e7-409a-a7e0-b8f7babc5aef'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:28.063522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-43546d3c-88e7-409a-a7e0-b8f7babc5aef admin admin] The token cache is being invalidate because project 2add46ff01ea465fb8275c01739ef48e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:57:28.063928 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-43546d3c-88e7-409a-a7e0-b8f7babc5aef admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:28.064825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-43546d3c-88e7-409a-a7e0-b8f7babc5aef admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:28.070588 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 901/1783] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:57:27 2022] DELETE /identity/v3/projects/2add46ff01ea465fb8275c01739ef48e => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:57:28.073098 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e611fc76-403d-4438-8300-325f894a8530 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:28.073413 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e611fc76-403d-4438-8300-325f894a8530 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:28.073697 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e611fc76-403d-4438-8300-325f894a8530 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:28.112437 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-e611fc76-403d-4438-8300-325f894a8530 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:28.117179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-e611fc76-403d-4438-8300-325f894a8530 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:28.168672 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43546d3c-88e7-409a-a7e0-b8f7babc5aef admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:28.172837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-43546d3c-88e7-409a-a7e0-b8f7babc5aef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:28.282804 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 883/1784] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:28.284998 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c815d8a-4d7e-4fb9-83fa-044155d52b7f None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:28.285365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c815d8a-4d7e-4fb9-83fa-044155d52b7f None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:28.285753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c815d8a-4d7e-4fb9-83fa-044155d52b7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:28.362624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43546d3c-88e7-409a-a7e0-b8f7babc5aef admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:28.369787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-4c815d8a-4d7e-4fb9-83fa-044155d52b7f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:28.372218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035c9666-f43d-4329-9a67-f36f2a0b48f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:28.372938 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-4c815d8a-4d7e-4fb9-83fa-044155d52b7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:28.373553 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035c9666-f43d-4329-9a67-f36f2a0b48f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:28.374233 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035c9666-f43d-4329-9a67-f36f2a0b48f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:28.375039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035c9666-f43d-4329-9a67-f36f2a0b48f3 admin admin] PATH_INFO: `/v3/users/5eb3ddf6498e4f0c9375c53ba0ed6be5` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:28.376696 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035c9666-f43d-4329-9a67-f36f2a0b48f3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5eb3ddf6498e4f0c9375c53ba0ed6be5)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:28.379256 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035c9666-f43d-4329-9a67-f36f2a0b48f3 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:28.406402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 884/1785] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:28.410171 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c815d8a-4d7e-4fb9-83fa-044155d52b7f None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:28.436109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c815d8a-4d7e-4fb9-83fa-044155d52b7f None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:28.451521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb3c615-056e-4c59-b7c5-18b26cc903ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:28.452257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb3c615-056e-4c59-b7c5-18b26cc903ea admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:28.452536 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb3c615-056e-4c59-b7c5-18b26cc903ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:28.452797 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb3c615-056e-4c59-b7c5-18b26cc903ea admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:28.456767 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb3c615-056e-4c59-b7c5-18b26cc903ea admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:57:28.457097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb3c615-056e-4c59-b7c5-18b26cc903ea admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:28.459592 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb3c615-056e-4c59-b7c5-18b26cc903ea admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:28.491819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 885/1786] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:57:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:57:28.496492 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb3c615-056e-4c59-b7c5-18b26cc903ea admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:28.518378 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb3c615-056e-4c59-b7c5-18b26cc903ea admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:28.534817 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77907e3-3439-4bca-8787-b2cfa8924d57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:28.535465 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77907e3-3439-4bca-8787-b2cfa8924d57 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:28.535744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77907e3-3439-4bca-8787-b2cfa8924d57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:28.536017 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77907e3-3439-4bca-8787-b2cfa8924d57 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:28.536774 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77907e3-3439-4bca-8787-b2cfa8924d57 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:28.538610 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-035c9666-f43d-4329-9a67-f36f2a0b48f3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5eb3ddf6498e4f0c9375c53ba0ed6be5'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:28.539778 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77907e3-3439-4bca-8787-b2cfa8924d57 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:28.556063 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-035c9666-f43d-4329-9a67-f36f2a0b48f3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5eb3ddf6498e4f0c9375c53ba0ed6be5'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:28.576502 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:28.582707 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-035c9666-f43d-4329-9a67-f36f2a0b48f3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5eb3ddf6498e4f0c9375c53ba0ed6be5'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:28.585327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-035c9666-f43d-4329-9a67-f36f2a0b48f3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5eb3ddf6498e4f0c9375c53ba0ed6be5'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:28.595044 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 886/1787] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:57:28 2022] POST /identity/v3/projects => generated 385 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:57:28.596604 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77907e3-3439-4bca-8787-b2cfa8924d57 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:28.618035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-035c9666-f43d-4329-9a67-f36f2a0b48f3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5eb3ddf6498e4f0c9375c53ba0ed6be5'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:28.632990 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 902/1788] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:57:28 2022] DELETE /identity/v3/users/5eb3ddf6498e4f0c9375c53ba0ed6be5 => generated 0 bytes in 465 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:28.637305 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1800bd3-ccf1-463b-bf71-b556524fe1ee None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:28.637662 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1800bd3-ccf1-463b-bf71-b556524fe1ee None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:28.638000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1800bd3-ccf1-463b-bf71-b556524fe1ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:28.686625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-f1800bd3-ccf1-463b-bf71-b556524fe1ee None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:28.688701 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-f1800bd3-ccf1-463b-bf71-b556524fe1ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:28.702413 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:28.718049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77907e3-3439-4bca-8787-b2cfa8924d57 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:28.796613 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 903/1789] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:28.798386 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1800bd3-ccf1-463b-bf71-b556524fe1ee None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:28.816377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-f1800bd3-ccf1-463b-bf71-b556524fe1ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:28.826488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c029333-5910-4d78-a9dc-bd1bb4e4e9e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:28.827228 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c029333-5910-4d78-a9dc-bd1bb4e4e9e0 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:28.828657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c029333-5910-4d78-a9dc-bd1bb4e4e9e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:28.828997 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c029333-5910-4d78-a9dc-bd1bb4e4e9e0 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:28.831116 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c029333-5910-4d78-a9dc-bd1bb4e4e9e0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:28.836746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c029333-5910-4d78-a9dc-bd1bb4e4e9e0 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:28.841027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1800bd3-ccf1-463b-bf71-b556524fe1ee None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:28.853978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3414fd0b-6531-49ed-a224-022fd2645058 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:28.854744 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3414fd0b-6531-49ed-a224-022fd2645058 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:28.855816 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3414fd0b-6531-49ed-a224-022fd2645058 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:28.856717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3414fd0b-6531-49ed-a224-022fd2645058 admin admin] PATH_INFO: `/v3/projects/15684cd8f29440599ea291fd7f9f42a7` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:28.859108 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3414fd0b-6531-49ed-a224-022fd2645058 admin admin] RBAC: Authorizing `identity:delete_project(project_id=15684cd8f29440599ea291fd7f9f42a7)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:28.863964 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3414fd0b-6531-49ed-a224-022fd2645058 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:28.887298 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 887/1790] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:57:28 2022] POST /identity/v3/users => generated 293 bytes in 288 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:57:28.887298 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c029333-5910-4d78-a9dc-bd1bb4e4e9e0 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:28.891333 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-0c029333-5910-4d78-a9dc-bd1bb4e4e9e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:28.943745 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-3414fd0b-6531-49ed-a224-022fd2645058 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '15684cd8f29440599ea291fd7f9f42a7', 'request_id': 'req-3414fd0b-6531-49ed-a224-022fd2645058'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:28.950601 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-3414fd0b-6531-49ed-a224-022fd2645058 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '15684cd8f29440599ea291fd7f9f42a7', 'request_id': 'req-3414fd0b-6531-49ed-a224-022fd2645058'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:28.958388 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-3414fd0b-6531-49ed-a224-022fd2645058 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '15684cd8f29440599ea291fd7f9f42a7', 'request_id': 'req-3414fd0b-6531-49ed-a224-022fd2645058'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:28.959545 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c029333-5910-4d78-a9dc-bd1bb4e4e9e0 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:28.972806 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-3414fd0b-6531-49ed-a224-022fd2645058 admin admin] The token cache is being invalidate because project 15684cd8f29440599ea291fd7f9f42a7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:57:28.973297 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-3414fd0b-6531-49ed-a224-022fd2645058 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:28.974070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-3414fd0b-6531-49ed-a224-022fd2645058 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:28.979366 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 904/1791] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:57:28 2022] DELETE /identity/v3/projects/15684cd8f29440599ea291fd7f9f42a7 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:28.980490 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3414fd0b-6531-49ed-a224-022fd2645058 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:28.982120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3414fd0b-6531-49ed-a224-022fd2645058 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:29.018442 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4635fce-7c62-4c68-a5c4-c7efb6878e8e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:29.019090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4635fce-7c62-4c68-a5c4-c7efb6878e8e service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:29.019396 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4635fce-7c62-4c68-a5c4-c7efb6878e8e service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:29.019667 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4635fce-7c62-4c68-a5c4-c7efb6878e8e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:29.022173 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-f4635fce-7c62-4c68-a5c4-c7efb6878e8e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:29.050284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3414fd0b-6531-49ed-a224-022fd2645058 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:29.055873 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de1131d-f73a-4d5b-afb9-d1a9bd9c9437 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:29.056546 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de1131d-f73a-4d5b-afb9-d1a9bd9c9437 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:29.056830 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de1131d-f73a-4d5b-afb9-d1a9bd9c9437 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:29.057099 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de1131d-f73a-4d5b-afb9-d1a9bd9c9437 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:29.057762 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de1131d-f73a-4d5b-afb9-d1a9bd9c9437 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:29.059415 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de1131d-f73a-4d5b-afb9-d1a9bd9c9437 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:29.071654 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 905/1792] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:28 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:57:29.074910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de1131d-f73a-4d5b-afb9-d1a9bd9c9437 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:29.074910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-2de1131d-f73a-4d5b-afb9-d1a9bd9c9437 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:29.083388 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:29.088478 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de1131d-f73a-4d5b-afb9-d1a9bd9c9437 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:29.093499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:29.094239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efdf6ff-6dbf-42ba-ab73-968661b874f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:29.095087 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efdf6ff-6dbf-42ba-ab73-968661b874f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:29.095377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efdf6ff-6dbf-42ba-ab73-968661b874f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:29.095889 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efdf6ff-6dbf-42ba-ab73-968661b874f8 admin admin] PATH_INFO: `/v3/users/c394cf1a20bf49a5b64741d64640cd08` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:29.096856 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4635fce-7c62-4c68-a5c4-c7efb6878e8e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:29.097098 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efdf6ff-6dbf-42ba-ab73-968661b874f8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c394cf1a20bf49a5b64741d64640cd08)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:29.098887 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4635fce-7c62-4c68-a5c4-c7efb6878e8e service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:29.099491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efdf6ff-6dbf-42ba-ab73-968661b874f8 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:29.109336 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 888/1793] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:29.110249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4635fce-7c62-4c68-a5c4-c7efb6878e8e service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:29.120237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4635fce-7c62-4c68-a5c4-c7efb6878e8e service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:29.124997 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e41580-2d8b-4afe-bf57-139510c0f4ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:29.125846 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e41580-2d8b-4afe-bf57-139510c0f4ea admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:29.126067 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e41580-2d8b-4afe-bf57-139510c0f4ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:29.126364 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e41580-2d8b-4afe-bf57-139510c0f4ea admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:29.127022 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e41580-2d8b-4afe-bf57-139510c0f4ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:29.128663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e41580-2d8b-4afe-bf57-139510c0f4ea admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:29.142289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 889/1794] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:29 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:57:29.147997 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e41580-2d8b-4afe-bf57-139510c0f4ea admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:29.158052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5efdf6ff-6dbf-42ba-ab73-968661b874f8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c394cf1a20bf49a5b64741d64640cd08'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:29.167786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e41580-2d8b-4afe-bf57-139510c0f4ea admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:29.171491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5efdf6ff-6dbf-42ba-ab73-968661b874f8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c394cf1a20bf49a5b64741d64640cd08'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:29.181579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6893a052-8ef4-44f8-8ac4-ba712d209a84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:29.186623 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6893a052-8ef4-44f8-8ac4-ba712d209a84 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:29.186917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6893a052-8ef4-44f8-8ac4-ba712d209a84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:29.187177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6893a052-8ef4-44f8-8ac4-ba712d209a84 admin admin] PATH_INFO: `/v3/projects/81b774e3b54b4fe3bf25502b2ec64484/users/a649b40eac36441bbcadb219a21a8907/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:29.199571 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5efdf6ff-6dbf-42ba-ab73-968661b874f8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c394cf1a20bf49a5b64741d64640cd08'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:29.200335 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5efdf6ff-6dbf-42ba-ab73-968661b874f8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c394cf1a20bf49a5b64741d64640cd08'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:29.211261 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5efdf6ff-6dbf-42ba-ab73-968661b874f8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c394cf1a20bf49a5b64741d64640cd08'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:29.215637 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6893a052-8ef4-44f8-8ac4-ba712d209a84 admin admin] RBAC: Authorizing `identity:create_grant(project_id=81b774e3b54b4fe3bf25502b2ec64484, user_id=a649b40eac36441bbcadb219a21a8907, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:29.219387 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6893a052-8ef4-44f8-8ac4-ba712d209a84 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:29.223597 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 906/1795] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:57:29 2022] DELETE /identity/v3/users/c394cf1a20bf49a5b64741d64640cd08 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:29.250761 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 890/1796] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:57:29 2022] PUT /identity/v3/projects/81b774e3b54b4fe3bf25502b2ec64484/users/a649b40eac36441bbcadb219a21a8907/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:29.264805 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212b3b10-1c23-4d56-a650-b2b783d6c763 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:29.265145 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212b3b10-1c23-4d56-a650-b2b783d6c763 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:29.265412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212b3b10-1c23-4d56-a650-b2b783d6c763 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:29.325870 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-212b3b10-1c23-4d56-a650-b2b783d6c763 None None] MFA Rules not processed for user `a649b40eac36441bbcadb219a21a8907`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:29.327290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-212b3b10-1c23-4d56-a650-b2b783d6c763 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:29.507729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 907/1797] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:29 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:29.517574 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6893a052-8ef4-44f8-8ac4-ba712d209a84 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:29.529217 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-6893a052-8ef4-44f8-8ac4-ba712d209a84 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:29.695249 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212b3b10-1c23-4d56-a650-b2b783d6c763 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:29.698879 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-212b3b10-1c23-4d56-a650-b2b783d6c763 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:29.748881 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6893a052-8ef4-44f8-8ac4-ba712d209a84 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:29.761909 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343147d8-3c7d-4f1c-b1dc-b893bf69b5bb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:29.762731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343147d8-3c7d-4f1c-b1dc-b893bf69b5bb service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:29.762926 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343147d8-3c7d-4f1c-b1dc-b893bf69b5bb service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:29.763187 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343147d8-3c7d-4f1c-b1dc-b893bf69b5bb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:29.790330 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343147d8-3c7d-4f1c-b1dc-b893bf69b5bb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:29.791162 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343147d8-3c7d-4f1c-b1dc-b893bf69b5bb service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:29.823747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 891/1798] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:29 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:29.833030 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212b3b10-1c23-4d56-a650-b2b783d6c763 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:29.842606 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fdcffd-1e74-4457-ab24-14103c2f916e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:29.844046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fdcffd-1e74-4457-ab24-14103c2f916e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:29.844694 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fdcffd-1e74-4457-ab24-14103c2f916e admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:29.845601 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fdcffd-1e74-4457-ab24-14103c2f916e admin admin] PATH_INFO: `/v3/projects/0ca870add6214f1394474d6201aef62b` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:29.847593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2fdcffd-1e74-4457-ab24-14103c2f916e admin admin] RBAC: Authorizing `identity:delete_project(project_id=0ca870add6214f1394474d6201aef62b)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:29.852024 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2fdcffd-1e74-4457-ab24-14103c2f916e admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:29.867639 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:29.979582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-b2fdcffd-1e74-4457-ab24-14103c2f916e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0ca870add6214f1394474d6201aef62b', 'request_id': 'req-b2fdcffd-1e74-4457-ab24-14103c2f916e'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:29.988999 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-b2fdcffd-1e74-4457-ab24-14103c2f916e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0ca870add6214f1394474d6201aef62b', 'request_id': 'req-b2fdcffd-1e74-4457-ab24-14103c2f916e'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:30.002398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-b2fdcffd-1e74-4457-ab24-14103c2f916e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0ca870add6214f1394474d6201aef62b', 'request_id': 'req-b2fdcffd-1e74-4457-ab24-14103c2f916e'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:30.033149 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-b2fdcffd-1e74-4457-ab24-14103c2f916e admin admin] The token cache is being invalidate because project 0ca870add6214f1394474d6201aef62b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:57:30.033532 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-b2fdcffd-1e74-4457-ab24-14103c2f916e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:30.037651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-b2fdcffd-1e74-4457-ab24-14103c2f916e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:30.042343 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 908/1799] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:57:29 2022] DELETE /identity/v3/projects/0ca870add6214f1394474d6201aef62b => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:57:30.046659 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343147d8-3c7d-4f1c-b1dc-b893bf69b5bb service cinder] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:30.068390 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343147d8-3c7d-4f1c-b1dc-b893bf69b5bb service cinder] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:30.228510 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06648216-d392-4aca-82da-33ac5a558d84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:30.229174 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06648216-d392-4aca-82da-33ac5a558d84 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:30.229455 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06648216-d392-4aca-82da-33ac5a558d84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:30.229730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06648216-d392-4aca-82da-33ac5a558d84 admin admin] PATH_INFO: `/v3/users/180e810b87724505821317da0f4cff7a` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:30.230922 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06648216-d392-4aca-82da-33ac5a558d84 admin admin] RBAC: Authorizing `identity:delete_user(user_id=180e810b87724505821317da0f4cff7a)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:30.232854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06648216-d392-4aca-82da-33ac5a558d84 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:30.362107 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-06648216-d392-4aca-82da-33ac5a558d84 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '180e810b87724505821317da0f4cff7a'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:30.377968 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fdcffd-1e74-4457-ab24-14103c2f916e admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:30.380281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-b2fdcffd-1e74-4457-ab24-14103c2f916e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:30.382637 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-06648216-d392-4aca-82da-33ac5a558d84 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '180e810b87724505821317da0f4cff7a'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:30.383766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-06648216-d392-4aca-82da-33ac5a558d84 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '180e810b87724505821317da0f4cff7a'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:30.384570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-06648216-d392-4aca-82da-33ac5a558d84 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '180e810b87724505821317da0f4cff7a'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:30.403802 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-06648216-d392-4aca-82da-33ac5a558d84 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '180e810b87724505821317da0f4cff7a'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:30.406498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:30.417901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fdcffd-1e74-4457-ab24-14103c2f916e admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:30.444905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c27faf9-aa27-48f6-b476-fcc6d499e914 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:30.447590 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c27faf9-aa27-48f6-b476-fcc6d499e914 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:30.448026 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c27faf9-aa27-48f6-b476-fcc6d499e914 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:30.448395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c27faf9-aa27-48f6-b476-fcc6d499e914 admin admin] PATH_INFO: `/v3/projects/73dfec534a5a419b9327c067863a160f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:30.449810 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c27faf9-aa27-48f6-b476-fcc6d499e914 admin admin] RBAC: Authorizing `identity:delete_project(project_id=73dfec534a5a419b9327c067863a160f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:30.452496 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c27faf9-aa27-48f6-b476-fcc6d499e914 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:30.453241 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 892/1800] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:57:30 2022] DELETE /identity/v3/users/180e810b87724505821317da0f4cff7a => generated 0 bytes in 407 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:57:30.622533 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2c27faf9-aa27-48f6-b476-fcc6d499e914 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '73dfec534a5a419b9327c067863a160f', 'request_id': 'req-2c27faf9-aa27-48f6-b476-fcc6d499e914'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:30.681465 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2c27faf9-aa27-48f6-b476-fcc6d499e914 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '73dfec534a5a419b9327c067863a160f', 'request_id': 'req-2c27faf9-aa27-48f6-b476-fcc6d499e914'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:30.711338 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2c27faf9-aa27-48f6-b476-fcc6d499e914 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '73dfec534a5a419b9327c067863a160f', 'request_id': 'req-2c27faf9-aa27-48f6-b476-fcc6d499e914'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:30.763347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2c27faf9-aa27-48f6-b476-fcc6d499e914 admin admin] The token cache is being invalidate because project 73dfec534a5a419b9327c067863a160f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:57:30.764228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2c27faf9-aa27-48f6-b476-fcc6d499e914 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:30.769281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2c27faf9-aa27-48f6-b476-fcc6d499e914 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:30.782548 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 909/1801] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:57:30 2022] DELETE /identity/v3/projects/73dfec534a5a419b9327c067863a160f => generated 0 bytes in 403 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:30.800274 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dde78a8-f293-4149-8208-41231e741d7a None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:30.801298 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dde78a8-f293-4149-8208-41231e741d7a None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:30.801646 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dde78a8-f293-4149-8208-41231e741d7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:30.861718 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-4dde78a8-f293-4149-8208-41231e741d7a None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:30.865141 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-4dde78a8-f293-4149-8208-41231e741d7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:31.027452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 893/1802] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:31.036142 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030b8230-5049-4af4-b8d8-9da291c3617f None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:31.036387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030b8230-5049-4af4-b8d8-9da291c3617f None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:31.036666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030b8230-5049-4af4-b8d8-9da291c3617f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:31.062923 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-030b8230-5049-4af4-b8d8-9da291c3617f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:31.064678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-030b8230-5049-4af4-b8d8-9da291c3617f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:31.066406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dde78a8-f293-4149-8208-41231e741d7a None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:31.069136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-4dde78a8-f293-4149-8208-41231e741d7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:31.078522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 910/1803] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:31.082513 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030b8230-5049-4af4-b8d8-9da291c3617f None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:31.092024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dde78a8-f293-4149-8208-41231e741d7a None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:31.093653 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030b8230-5049-4af4-b8d8-9da291c3617f None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:31.099072 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c77249-8b65-40a5-a759-e9684b9db245 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:31.099896 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c77249-8b65-40a5-a759-e9684b9db245 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:31.100105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f880a40-bcb3-4c3a-a31d-4c4a9fc10af0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:31.100560 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c77249-8b65-40a5-a759-e9684b9db245 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:31.100783 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f880a40-bcb3-4c3a-a31d-4c4a9fc10af0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:31.101073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f880a40-bcb3-4c3a-a31d-4c4a9fc10af0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:31.101285 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c77249-8b65-40a5-a759-e9684b9db245 admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:31.101425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f880a40-bcb3-4c3a-a31d-4c4a9fc10af0 admin admin] PATH_INFO: `/v3/projects/51fde31499574e58a2d0d72ebb521a68` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:31.102542 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c77249-8b65-40a5-a759-e9684b9db245 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:57:31.103000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c77249-8b65-40a5-a759-e9684b9db245 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:31.105481 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c77249-8b65-40a5-a759-e9684b9db245 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:31.112348 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f880a40-bcb3-4c3a-a31d-4c4a9fc10af0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=51fde31499574e58a2d0d72ebb521a68)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:31.114174 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f880a40-bcb3-4c3a-a31d-4c4a9fc10af0 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:31.120141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 911/1804] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:57:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:57:31.124460 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c77249-8b65-40a5-a759-e9684b9db245 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:31.136115 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c77249-8b65-40a5-a759-e9684b9db245 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:31.156124 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2554bc6b-6a92-4beb-b47e-677c7c8bcc50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:31.156809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2554bc6b-6a92-4beb-b47e-677c7c8bcc50 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:31.157020 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2554bc6b-6a92-4beb-b47e-677c7c8bcc50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:31.157298 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2554bc6b-6a92-4beb-b47e-677c7c8bcc50 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:31.157938 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2554bc6b-6a92-4beb-b47e-677c7c8bcc50 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:31.159895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2554bc6b-6a92-4beb-b47e-677c7c8bcc50 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:31.176062 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6f880a40-bcb3-4c3a-a31d-4c4a9fc10af0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '51fde31499574e58a2d0d72ebb521a68', 'request_id': 'req-6f880a40-bcb3-4c3a-a31d-4c4a9fc10af0'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:31.190503 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6f880a40-bcb3-4c3a-a31d-4c4a9fc10af0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '51fde31499574e58a2d0d72ebb521a68', 'request_id': 'req-6f880a40-bcb3-4c3a-a31d-4c4a9fc10af0'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:31.204509 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6f880a40-bcb3-4c3a-a31d-4c4a9fc10af0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '51fde31499574e58a2d0d72ebb521a68', 'request_id': 'req-6f880a40-bcb3-4c3a-a31d-4c4a9fc10af0'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:31.232694 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 912/1805] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:57:31 2022] POST /identity/v3/projects => generated 385 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:57:31.234750 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6f880a40-bcb3-4c3a-a31d-4c4a9fc10af0 admin admin] The token cache is being invalidate because project 51fde31499574e58a2d0d72ebb521a68 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:57:31.235046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6f880a40-bcb3-4c3a-a31d-4c4a9fc10af0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:31.236521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2554bc6b-6a92-4beb-b47e-677c7c8bcc50 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:31.237195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6f880a40-bcb3-4c3a-a31d-4c4a9fc10af0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:31.242962 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 894/1806] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:57:31 2022] DELETE /identity/v3/projects/51fde31499574e58a2d0d72ebb521a68 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:31.245530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c47fc0-f220-461c-8ba9-ab1573fdba7a None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:31.245816 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c47fc0-f220-461c-8ba9-ab1573fdba7a None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:31.246087 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c47fc0-f220-461c-8ba9-ab1573fdba7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:31.272188 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-14c47fc0-f220-461c-8ba9-ab1573fdba7a None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:31.273683 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-14c47fc0-f220-461c-8ba9-ab1573fdba7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:31.301826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:31.333231 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2554bc6b-6a92-4beb-b47e-677c7c8bcc50 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:31.340757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5f7914-0be1-4bf9-8600-a02fbc83bf10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:31.342936 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5f7914-0be1-4bf9-8600-a02fbc83bf10 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:31.343623 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5f7914-0be1-4bf9-8600-a02fbc83bf10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:31.344349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5f7914-0be1-4bf9-8600-a02fbc83bf10 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:31.344563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5f7914-0be1-4bf9-8600-a02fbc83bf10 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:31.346756 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5f7914-0be1-4bf9-8600-a02fbc83bf10 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:31.355210 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 895/1807] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:31.366996 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013c34a3-4d82-484e-9630-f82fb103e974 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:31.367504 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013c34a3-4d82-484e-9630-f82fb103e974 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:31.368189 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013c34a3-4d82-484e-9630-f82fb103e974 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:31.375711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 913/1808] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:57:31 2022] POST /identity/v3/users => generated 293 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:57:31.380006 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5f7914-0be1-4bf9-8600-a02fbc83bf10 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:31.382444 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-fa5f7914-0be1-4bf9-8600-a02fbc83bf10 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:31.391545 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:31.396842 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5f7914-0be1-4bf9-8600-a02fbc83bf10 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:31.400348 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-013c34a3-4d82-484e-9630-f82fb103e974 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:31.401786 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248cc349-b328-43af-9736-c38a2af6488f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:31.402133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-013c34a3-4d82-484e-9630-f82fb103e974 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:31.402390 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248cc349-b328-43af-9736-c38a2af6488f admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:31.402708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248cc349-b328-43af-9736-c38a2af6488f admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:31.402982 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248cc349-b328-43af-9736-c38a2af6488f admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:31.403615 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248cc349-b328-43af-9736-c38a2af6488f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:31.405206 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248cc349-b328-43af-9736-c38a2af6488f admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:31.410372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 896/1809] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:31.414851 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013c34a3-4d82-484e-9630-f82fb103e974 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:31.419578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 914/1810] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:31 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:57:31.420615 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248cc349-b328-43af-9736-c38a2af6488f admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:31.424145 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013c34a3-4d82-484e-9630-f82fb103e974 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:31.429561 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e90fe03-93d2-40b3-b360-7ef250ef4f93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:31.430282 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e90fe03-93d2-40b3-b360-7ef250ef4f93 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:31.430537 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248cc349-b328-43af-9736-c38a2af6488f admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:31.430634 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e90fe03-93d2-40b3-b360-7ef250ef4f93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:31.430868 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e90fe03-93d2-40b3-b360-7ef250ef4f93 admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:31.431523 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e90fe03-93d2-40b3-b360-7ef250ef4f93 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:57:31.431829 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e90fe03-93d2-40b3-b360-7ef250ef4f93 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:31.433429 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e90fe03-93d2-40b3-b360-7ef250ef4f93 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:31.435033 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70d90da-2e72-469b-b974-50aad44732c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:31.435633 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70d90da-2e72-469b-b974-50aad44732c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:31.435913 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70d90da-2e72-469b-b974-50aad44732c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:31.436185 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70d90da-2e72-469b-b974-50aad44732c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:31.436818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70d90da-2e72-469b-b974-50aad44732c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:31.438404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70d90da-2e72-469b-b974-50aad44732c4 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:31.447006 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 897/1811] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:57:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:57:31.453173 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 915/1812] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:31 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:57:31.453335 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e90fe03-93d2-40b3-b360-7ef250ef4f93 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:31.454187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70d90da-2e72-469b-b974-50aad44732c4 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:31.462318 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e90fe03-93d2-40b3-b360-7ef250ef4f93 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:31.465050 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70d90da-2e72-469b-b974-50aad44732c4 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:31.467926 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f9d730-43fe-4ab4-8289-02eb067b7311 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:31.468722 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f9d730-43fe-4ab4-8289-02eb067b7311 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:31.469071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f9d730-43fe-4ab4-8289-02eb067b7311 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:31.469427 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f9d730-43fe-4ab4-8289-02eb067b7311 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:31.469538 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8186f127-02e3-4ddd-820c-e293c1641ee7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:31.470170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8186f127-02e3-4ddd-820c-e293c1641ee7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:31.470286 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f9d730-43fe-4ab4-8289-02eb067b7311 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:31.470448 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8186f127-02e3-4ddd-820c-e293c1641ee7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:31.470742 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8186f127-02e3-4ddd-820c-e293c1641ee7 admin admin] PATH_INFO: `/v3/projects/a4029e46d5224f7d9b3a4f57a017003d/users/fc194778cd604c46b4e59bc060f53315/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:31.472453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f9d730-43fe-4ab4-8289-02eb067b7311 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:31.484992 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8186f127-02e3-4ddd-820c-e293c1641ee7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a4029e46d5224f7d9b3a4f57a017003d, user_id=fc194778cd604c46b4e59bc060f53315, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:31.487737 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8186f127-02e3-4ddd-820c-e293c1641ee7 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:31.504101 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 898/1813] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:57:31 2022] POST /identity/v3/projects => generated 363 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:57:31.505129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f9d730-43fe-4ab4-8289-02eb067b7311 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:31.506448 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 916/1814] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:57:31 2022] PUT /identity/v3/projects/a4029e46d5224f7d9b3a4f57a017003d/users/fc194778cd604c46b4e59bc060f53315/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:31.516646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da319432-f5e8-4a6b-b55e-a3462cfdcdbc None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:31.516977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da319432-f5e8-4a6b-b55e-a3462cfdcdbc None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:31.517289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da319432-f5e8-4a6b-b55e-a3462cfdcdbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:31.549229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-da319432-f5e8-4a6b-b55e-a3462cfdcdbc None None] MFA Rules not processed for user `fc194778cd604c46b4e59bc060f53315`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:31.550763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-da319432-f5e8-4a6b-b55e-a3462cfdcdbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:31.584400 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f9d730-43fe-4ab4-8289-02eb067b7311 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:31.590546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5595d2-5b23-46b5-a009-ec9d030e70e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:31.591109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5595d2-5b23-46b5-a009-ec9d030e70e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:31.591389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5595d2-5b23-46b5-a009-ec9d030e70e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:31.591754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5595d2-5b23-46b5-a009-ec9d030e70e9 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:31.592702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5595d2-5b23-46b5-a009-ec9d030e70e9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:31.594330 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5595d2-5b23-46b5-a009-ec9d030e70e9 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:31.619179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 899/1815] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:57:31 2022] POST /identity/v3/users => generated 282 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:57:31.623503 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5595d2-5b23-46b5-a009-ec9d030e70e9 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:31.636920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 917/1816] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:31 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:57:31.637601 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5595d2-5b23-46b5-a009-ec9d030e70e9 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:31.645813 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfb5f79-c50d-4efe-a4bb-fec70555486a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:31.647012 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfb5f79-c50d-4efe-a4bb-fec70555486a admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:31.647012 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfb5f79-c50d-4efe-a4bb-fec70555486a admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:31.647203 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfb5f79-c50d-4efe-a4bb-fec70555486a admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:31.647889 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdfb5f79-c50d-4efe-a4bb-fec70555486a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:31.649480 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da319432-f5e8-4a6b-b55e-a3462cfdcdbc None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:31.649678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdfb5f79-c50d-4efe-a4bb-fec70555486a admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:31.650958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-da319432-f5e8-4a6b-b55e-a3462cfdcdbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:31.661988 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 900/1817] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:31 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:57:31.666019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfb5f79-c50d-4efe-a4bb-fec70555486a admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:31.675706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfb5f79-c50d-4efe-a4bb-fec70555486a admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:31.681240 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cdb389-0e48-49ee-8213-79ed2b8f2341 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:31.681874 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cdb389-0e48-49ee-8213-79ed2b8f2341 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:31.682158 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cdb389-0e48-49ee-8213-79ed2b8f2341 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:31.682436 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cdb389-0e48-49ee-8213-79ed2b8f2341 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:31.683244 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97cdb389-0e48-49ee-8213-79ed2b8f2341 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:31.684857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97cdb389-0e48-49ee-8213-79ed2b8f2341 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:31.697501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 901/1818] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:31 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:57:31.701228 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cdb389-0e48-49ee-8213-79ed2b8f2341 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:31.711133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cdb389-0e48-49ee-8213-79ed2b8f2341 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:31.715896 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3870dbed-4304-4f12-9ca7-5f3e4cde646c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:31.716562 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3870dbed-4304-4f12-9ca7-5f3e4cde646c admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:31.716842 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3870dbed-4304-4f12-9ca7-5f3e4cde646c admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:31.717120 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3870dbed-4304-4f12-9ca7-5f3e4cde646c admin admin] PATH_INFO: `/v3/projects/ee5ad84a77b74cf9ab2d9bc7959fb838/users/51a2ade99d324a07a6e350ee8ef28a77/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:31.717224 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da319432-f5e8-4a6b-b55e-a3462cfdcdbc None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:31.721195 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67d2505-3186-429f-94df-fb6344c8019a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:31.721886 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67d2505-3186-429f-94df-fb6344c8019a service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:31.722236 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67d2505-3186-429f-94df-fb6344c8019a service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:31.722616 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67d2505-3186-429f-94df-fb6344c8019a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:31.731800 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67d2505-3186-429f-94df-fb6344c8019a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:31.733342 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67d2505-3186-429f-94df-fb6344c8019a service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:31.734226 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3870dbed-4304-4f12-9ca7-5f3e4cde646c admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee5ad84a77b74cf9ab2d9bc7959fb838, user_id=51a2ade99d324a07a6e350ee8ef28a77, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:31.737169 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3870dbed-4304-4f12-9ca7-5f3e4cde646c admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:31.749315 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 918/1819] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:31 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:57:31.766507 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 902/1820] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:57:31 2022] PUT /identity/v3/projects/ee5ad84a77b74cf9ab2d9bc7959fb838/users/51a2ade99d324a07a6e350ee8ef28a77/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:57:31.769415 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67d2505-3186-429f-94df-fb6344c8019a service cinder] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:31.769918 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e12d615-a145-4eb1-b558-f65e50405345 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:31.770204 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e12d615-a145-4eb1-b558-f65e50405345 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:31.770507 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e12d615-a145-4eb1-b558-f65e50405345 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:31.780090 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67d2505-3186-429f-94df-fb6344c8019a service cinder] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:31.788806 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:31.798711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-4e12d615-a145-4eb1-b558-f65e50405345 None None] MFA Rules not processed for user `51a2ade99d324a07a6e350ee8ef28a77`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:31.800005 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-4e12d615-a145-4eb1-b558-f65e50405345 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:31.843601 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501d0243-6153-4617-a112-276d23694524 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:31.844316 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501d0243-6153-4617-a112-276d23694524 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:31.844665 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501d0243-6153-4617-a112-276d23694524 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:31.845006 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501d0243-6153-4617-a112-276d23694524 admin admin] PATH_INFO: `/v3/users/fc194778cd604c46b4e59bc060f53315` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:31.846215 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501d0243-6153-4617-a112-276d23694524 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fc194778cd604c46b4e59bc060f53315)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:31.848229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501d0243-6153-4617-a112-276d23694524 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:31.875940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:31.882743 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 903/1821] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:31 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:31.887449 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e12d615-a145-4eb1-b558-f65e50405345 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:31.897230 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-501d0243-6153-4617-a112-276d23694524 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fc194778cd604c46b4e59bc060f53315'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:31.897732 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e12d615-a145-4eb1-b558-f65e50405345 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:31.902787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd1c49d-75a1-47fa-84c3-4407b2b9157b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:31.904130 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd1c49d-75a1-47fa-84c3-4407b2b9157b admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:31.904406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd1c49d-75a1-47fa-84c3-4407b2b9157b admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:31.904678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd1c49d-75a1-47fa-84c3-4407b2b9157b admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:31.905211 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-501d0243-6153-4617-a112-276d23694524 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fc194778cd604c46b4e59bc060f53315'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:31.905341 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd1c49d-75a1-47fa-84c3-4407b2b9157b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:31.907315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd1c49d-75a1-47fa-84c3-4407b2b9157b admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:31.918945 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-501d0243-6153-4617-a112-276d23694524 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fc194778cd604c46b4e59bc060f53315'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:31.919652 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-501d0243-6153-4617-a112-276d23694524 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fc194778cd604c46b4e59bc060f53315'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:31.926752 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-501d0243-6153-4617-a112-276d23694524 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fc194778cd604c46b4e59bc060f53315'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:31.933604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 919/1822] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:57:31 2022] DELETE /identity/v3/users/fc194778cd604c46b4e59bc060f53315 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:31.939100 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 904/1823] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:57:31 2022] POST /identity/v3/projects => generated 365 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:57:31.939233 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef82796f-a842-4414-a98d-f2a9737d401f None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:31.939233 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef82796f-a842-4414-a98d-f2a9737d401f None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:31.939233 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef82796f-a842-4414-a98d-f2a9737d401f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:31.940151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd1c49d-75a1-47fa-84c3-4407b2b9157b admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:31.941793 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8fd1c49d-75a1-47fa-84c3-4407b2b9157b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:31.972166 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-ef82796f-a842-4414-a98d-f2a9737d401f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:31.973517 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-ef82796f-a842-4414-a98d-f2a9737d401f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:32.023908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd1c49d-75a1-47fa-84c3-4407b2b9157b admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:32.046159 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f62efa-f918-4811-bf1c-7483f2d56479 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:32.046854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f62efa-f918-4811-bf1c-7483f2d56479 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:32.047126 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f62efa-f918-4811-bf1c-7483f2d56479 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:32.047543 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f62efa-f918-4811-bf1c-7483f2d56479 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:32.048188 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f62efa-f918-4811-bf1c-7483f2d56479 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:32.050218 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f62efa-f918-4811-bf1c-7483f2d56479 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:32.067494 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 920/1824] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:57:32.075631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 905/1825] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:57:31 2022] POST /identity/v3/users => generated 289 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:57:32.078630 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef82796f-a842-4414-a98d-f2a9737d401f None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:32.079446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f62efa-f918-4811-bf1c-7483f2d56479 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:32.080358 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-ef82796f-a842-4414-a98d-f2a9737d401f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:32.090047 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f62efa-f918-4811-bf1c-7483f2d56479 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:32.094919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db84523-bae9-4cb5-9429-265e8cf9d8e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:32.095584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db84523-bae9-4cb5-9429-265e8cf9d8e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:32.095891 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db84523-bae9-4cb5-9429-265e8cf9d8e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:32.096178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db84523-bae9-4cb5-9429-265e8cf9d8e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:32.096973 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db84523-bae9-4cb5-9429-265e8cf9d8e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:32.098715 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db84523-bae9-4cb5-9429-265e8cf9d8e6 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:32.110586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 906/1826] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:32 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:57:32.115827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db84523-bae9-4cb5-9429-265e8cf9d8e6 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:32.126886 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db84523-bae9-4cb5-9429-265e8cf9d8e6 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:32.133603 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c5c0ab-9f87-4156-9ffd-94ce7c45b27a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:32.134282 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c5c0ab-9f87-4156-9ffd-94ce7c45b27a admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:32.134584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c5c0ab-9f87-4156-9ffd-94ce7c45b27a admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:32.134861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c5c0ab-9f87-4156-9ffd-94ce7c45b27a admin admin] PATH_INFO: `/v3/projects/cffe27e20147449e873e664ea9e4816c/users/898c445d3f1d40fc945fa70899aaf3b9/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:32.135565 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:32.152117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c5c0ab-9f87-4156-9ffd-94ce7c45b27a admin admin] RBAC: Authorizing `identity:create_grant(project_id=cffe27e20147449e873e664ea9e4816c, user_id=898c445d3f1d40fc945fa70899aaf3b9, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:32.154823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c5c0ab-9f87-4156-9ffd-94ce7c45b27a admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:32.164104 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef82796f-a842-4414-a98d-f2a9737d401f None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:32.170238 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 907/1827] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:57:32 2022] PUT /identity/v3/projects/cffe27e20147449e873e664ea9e4816c/users/898c445d3f1d40fc945fa70899aaf3b9/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:32.175444 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c5c0ab-9f87-4156-9ffd-94ce7c45b27a admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:32.218372 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0129248-79aa-4c9c-a354-f26020c9bb92 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:32.219169 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0129248-79aa-4c9c-a354-f26020c9bb92 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:32.219449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0129248-79aa-4c9c-a354-f26020c9bb92 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:32.219718 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0129248-79aa-4c9c-a354-f26020c9bb92 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:32.250801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c5c0ab-9f87-4156-9ffd-94ce7c45b27a admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:32.255903 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c702529-a8e5-4984-9764-ccaaa861e729 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:32.256614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c702529-a8e5-4984-9764-ccaaa861e729 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:32.256992 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c702529-a8e5-4984-9764-ccaaa861e729 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:32.257286 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c702529-a8e5-4984-9764-ccaaa861e729 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:32.258040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c702529-a8e5-4984-9764-ccaaa861e729 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:32.259792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c702529-a8e5-4984-9764-ccaaa861e729 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:32.271548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 908/1828] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:32 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:57:32.275962 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c702529-a8e5-4984-9764-ccaaa861e729 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:32.285779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c702529-a8e5-4984-9764-ccaaa861e729 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:32.291924 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4e6160-54f7-459e-815c-06b05cedb00f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:32.292619 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4e6160-54f7-459e-815c-06b05cedb00f admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:32.292907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4e6160-54f7-459e-815c-06b05cedb00f admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:32.293035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0129248-79aa-4c9c-a354-f26020c9bb92 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:32.293172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4e6160-54f7-459e-815c-06b05cedb00f admin admin] PATH_INFO: `/v3/projects/cffe27e20147449e873e664ea9e4816c/users/898c445d3f1d40fc945fa70899aaf3b9/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:32.294408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0129248-79aa-4c9c-a354-f26020c9bb92 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:32.295288 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4e6160-54f7-459e-815c-06b05cedb00f admin admin] RBAC: Authorizing `identity:create_grant(project_id=cffe27e20147449e873e664ea9e4816c, user_id=898c445d3f1d40fc945fa70899aaf3b9, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:32.297969 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4e6160-54f7-459e-815c-06b05cedb00f admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:32.306139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 921/1829] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 228 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:32.309441 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 909/1830] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:57:32 2022] PUT /identity/v3/projects/cffe27e20147449e873e664ea9e4816c/users/898c445d3f1d40fc945fa70899aaf3b9/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:57:32.323221 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0074a8-4f63-4d91-89cc-d886dd39a083 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:32.323741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0074a8-4f63-4d91-89cc-d886dd39a083 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:32.324028 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0074a8-4f63-4d91-89cc-d886dd39a083 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:32.359847 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-8b0074a8-4f63-4d91-89cc-d886dd39a083 None None] MFA Rules not processed for user `898c445d3f1d40fc945fa70899aaf3b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:32.361217 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-8b0074a8-4f63-4d91-89cc-d886dd39a083 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:32.462846 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 922/1831] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:32 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:57:32.481151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4e6160-54f7-459e-815c-06b05cedb00f admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:32.483014 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-2a4e6160-54f7-459e-815c-06b05cedb00f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:32.503761 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:32.561059 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4e6160-54f7-459e-815c-06b05cedb00f admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:32.566977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b34c991-26b7-4174-901c-9bfcde404197 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:32.567601 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b34c991-26b7-4174-901c-9bfcde404197 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:32.568384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b34c991-26b7-4174-901c-9bfcde404197 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:32.568662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b34c991-26b7-4174-901c-9bfcde404197 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:32.579788 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b34c991-26b7-4174-901c-9bfcde404197 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:32.581236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b34c991-26b7-4174-901c-9bfcde404197 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:32.594129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 910/1832] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:32 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:32.615060 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0074a8-4f63-4d91-89cc-d886dd39a083 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:32.616546 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-8b0074a8-4f63-4d91-89cc-d886dd39a083 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:32.672456 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:32.688972 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b34c991-26b7-4174-901c-9bfcde404197 service cinder] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:32.698923 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b34c991-26b7-4174-901c-9bfcde404197 service cinder] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:32.702992 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5483b484-5f1a-481f-abd2-5f715f849146 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:32.703678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5483b484-5f1a-481f-abd2-5f715f849146 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:32.704379 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5483b484-5f1a-481f-abd2-5f715f849146 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:32.704724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5483b484-5f1a-481f-abd2-5f715f849146 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:32.706699 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-5483b484-5f1a-481f-abd2-5f715f849146 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:32.716295 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0074a8-4f63-4d91-89cc-d886dd39a083 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:32.721029 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f7cef7-8945-451d-9728-e147e1fc1343 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:32.721666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f7cef7-8945-451d-9728-e147e1fc1343 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:32.721946 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f7cef7-8945-451d-9728-e147e1fc1343 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:32.722343 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f7cef7-8945-451d-9728-e147e1fc1343 admin admin] PATH_INFO: `/v3/projects/a4029e46d5224f7d9b3a4f57a017003d` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:32.723636 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f7cef7-8945-451d-9728-e147e1fc1343 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a4029e46d5224f7d9b3a4f57a017003d)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:32.725588 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f7cef7-8945-451d-9728-e147e1fc1343 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:32.770785 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5483b484-5f1a-481f-abd2-5f715f849146 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:32.772839 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5483b484-5f1a-481f-abd2-5f715f849146 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:32.776493 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-b3f7cef7-8945-451d-9728-e147e1fc1343 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a4029e46d5224f7d9b3a4f57a017003d', 'request_id': 'req-b3f7cef7-8945-451d-9728-e147e1fc1343'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:32.784390 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-b3f7cef7-8945-451d-9728-e147e1fc1343 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a4029e46d5224f7d9b3a4f57a017003d', 'request_id': 'req-b3f7cef7-8945-451d-9728-e147e1fc1343'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:32.784600 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 911/1833] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:32 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:57:32.794492 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-b3f7cef7-8945-451d-9728-e147e1fc1343 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a4029e46d5224f7d9b3a4f57a017003d', 'request_id': 'req-b3f7cef7-8945-451d-9728-e147e1fc1343'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:32.808699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-b3f7cef7-8945-451d-9728-e147e1fc1343 admin admin] The token cache is being invalidate because project a4029e46d5224f7d9b3a4f57a017003d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:57:32.809314 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-b3f7cef7-8945-451d-9728-e147e1fc1343 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:32.809980 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-b3f7cef7-8945-451d-9728-e147e1fc1343 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:32.811815 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 923/1834] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:57:32 2022] DELETE /identity/v3/projects/a4029e46d5224f7d9b3a4f57a017003d => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:57:32.816761 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d25dd7b-38cd-442a-a38b-237c6a69c3fd None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:32.817055 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d25dd7b-38cd-442a-a38b-237c6a69c3fd None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:32.817320 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d25dd7b-38cd-442a-a38b-237c6a69c3fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:32.842230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-8d25dd7b-38cd-442a-a38b-237c6a69c3fd None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:32.843570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8d25dd7b-38cd-442a-a38b-237c6a69c3fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:32.940490 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 912/1835] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:32.949881 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f687e95f-94de-4297-99ac-fd5733a3c157 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:32.950252 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f687e95f-94de-4297-99ac-fd5733a3c157 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:32.950658 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f687e95f-94de-4297-99ac-fd5733a3c157 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:32.972998 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-f687e95f-94de-4297-99ac-fd5733a3c157 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:32.974258 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-f687e95f-94de-4297-99ac-fd5733a3c157 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:32.982621 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 924/1836] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:32.986795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d25dd7b-38cd-442a-a38b-237c6a69c3fd None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:32.997200 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d25dd7b-38cd-442a-a38b-237c6a69c3fd None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:33.001822 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a526721d-f840-4626-9afc-525e7bb36298 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:33.002466 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a526721d-f840-4626-9afc-525e7bb36298 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:33.002776 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a526721d-f840-4626-9afc-525e7bb36298 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:33.003123 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a526721d-f840-4626-9afc-525e7bb36298 admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:33.003980 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a526721d-f840-4626-9afc-525e7bb36298 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:57:33.004343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a526721d-f840-4626-9afc-525e7bb36298 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:33.006875 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a526721d-f840-4626-9afc-525e7bb36298 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:33.020181 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 913/1837] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:57:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:57:33.024704 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f687e95f-94de-4297-99ac-fd5733a3c157 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:33.034736 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f687e95f-94de-4297-99ac-fd5733a3c157 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:33.041357 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e1d5ec-6f72-426a-aea8-1f6ded2e3efb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:33.042173 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e1d5ec-6f72-426a-aea8-1f6ded2e3efb admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:33.042606 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e1d5ec-6f72-426a-aea8-1f6ded2e3efb admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:33.042973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e1d5ec-6f72-426a-aea8-1f6ded2e3efb admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:33.043699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e1d5ec-6f72-426a-aea8-1f6ded2e3efb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:33.051962 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e1d5ec-6f72-426a-aea8-1f6ded2e3efb admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:33.080226 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 925/1838] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:57:33 2022] POST /identity/v3/projects => generated 387 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:57:33.086157 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a526721d-f840-4626-9afc-525e7bb36298 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:33.098678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a526721d-f840-4626-9afc-525e7bb36298 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:33.155768 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:33.184041 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891aa4ed-c75b-4f9f-a24b-51e1f8cf1c7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:33.185971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891aa4ed-c75b-4f9f-a24b-51e1f8cf1c7c admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:33.186327 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891aa4ed-c75b-4f9f-a24b-51e1f8cf1c7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:33.186704 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891aa4ed-c75b-4f9f-a24b-51e1f8cf1c7c admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:33.187424 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891aa4ed-c75b-4f9f-a24b-51e1f8cf1c7c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:33.189567 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891aa4ed-c75b-4f9f-a24b-51e1f8cf1c7c admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:33.222196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 914/1839] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:57:33 2022] POST /identity/v3/users => generated 294 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:57:33.224103 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e1d5ec-6f72-426a-aea8-1f6ded2e3efb admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:33.235609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e1d5ec-6f72-426a-aea8-1f6ded2e3efb admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:33.242202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1859a5-24ce-40aa-a8cd-f29c045616ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:33.242867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1859a5-24ce-40aa-a8cd-f29c045616ba admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:33.243178 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1859a5-24ce-40aa-a8cd-f29c045616ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:33.243448 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1859a5-24ce-40aa-a8cd-f29c045616ba admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:33.244416 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1859a5-24ce-40aa-a8cd-f29c045616ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:33.245757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1859a5-24ce-40aa-a8cd-f29c045616ba admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:33.257425 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 926/1840] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:33 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:57:33.261624 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891aa4ed-c75b-4f9f-a24b-51e1f8cf1c7c admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:33.273606 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891aa4ed-c75b-4f9f-a24b-51e1f8cf1c7c admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:33.278368 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b44382-4b2a-48e6-972e-0a020b0c901e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:33.279144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b44382-4b2a-48e6-972e-0a020b0c901e admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:33.279400 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b44382-4b2a-48e6-972e-0a020b0c901e admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:33.279535 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b44382-4b2a-48e6-972e-0a020b0c901e admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:33.280155 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b44382-4b2a-48e6-972e-0a020b0c901e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:33.281773 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b44382-4b2a-48e6-972e-0a020b0c901e admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:33.294789 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 915/1841] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:33 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:57:33.299058 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1859a5-24ce-40aa-a8cd-f29c045616ba admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:33.317211 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1859a5-24ce-40aa-a8cd-f29c045616ba admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:33.323185 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce57939a-ec75-4f8a-b2ca-8a4728bbda09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:33.323986 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce57939a-ec75-4f8a-b2ca-8a4728bbda09 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:33.324271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce57939a-ec75-4f8a-b2ca-8a4728bbda09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:33.324548 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce57939a-ec75-4f8a-b2ca-8a4728bbda09 admin admin] PATH_INFO: `/v3/projects/454d03699de9403c9c3f7b05e7b5609e/users/ed1c9b87ade9484fa37fe91bf1133b21/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:33.340532 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce57939a-ec75-4f8a-b2ca-8a4728bbda09 admin admin] RBAC: Authorizing `identity:create_grant(project_id=454d03699de9403c9c3f7b05e7b5609e, user_id=ed1c9b87ade9484fa37fe91bf1133b21, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:33.343246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce57939a-ec75-4f8a-b2ca-8a4728bbda09 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:33.364438 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 927/1842] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:57:33 2022] PUT /identity/v3/projects/454d03699de9403c9c3f7b05e7b5609e/users/ed1c9b87ade9484fa37fe91bf1133b21/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:57:33.377785 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e76bae-c1eb-4356-887d-1b6b74ec3194 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:33.382739 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e76bae-c1eb-4356-887d-1b6b74ec3194 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:33.383267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e76bae-c1eb-4356-887d-1b6b74ec3194 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:33.429728 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-e6e76bae-c1eb-4356-887d-1b6b74ec3194 None None] MFA Rules not processed for user `ed1c9b87ade9484fa37fe91bf1133b21`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:33.431163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-e6e76bae-c1eb-4356-887d-1b6b74ec3194 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:33.601159 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 916/1843] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:33 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:33.609777 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce57939a-ec75-4f8a-b2ca-8a4728bbda09 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:33.613549 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-ce57939a-ec75-4f8a-b2ca-8a4728bbda09 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:33.773636 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce57939a-ec75-4f8a-b2ca-8a4728bbda09 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:33.780048 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b06792-30a8-4885-9e41-a8eefaf36f0a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:33.780804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b06792-30a8-4885-9e41-a8eefaf36f0a service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:33.781099 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b06792-30a8-4885-9e41-a8eefaf36f0a service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:33.781428 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b06792-30a8-4885-9e41-a8eefaf36f0a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:33.790018 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:33.799886 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b06792-30a8-4885-9e41-a8eefaf36f0a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:33.801667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b06792-30a8-4885-9e41-a8eefaf36f0a service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:33.816091 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 928/1844] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:33 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:43.577856 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=47,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:43.808930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:49.944678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e76bae-c1eb-4356-887d-1b6b74ec3194 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:49.946164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-e6e76bae-c1eb-4356-887d-1b6b74ec3194 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:50.101006 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e76bae-c1eb-4356-887d-1b6b74ec3194 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:50.109743 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b461196-2b6a-4bc4-b593-af4ad4b84929 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:50.110411 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b461196-2b6a-4bc4-b593-af4ad4b84929 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:50.110794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b461196-2b6a-4bc4-b593-af4ad4b84929 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:50.111065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b461196-2b6a-4bc4-b593-af4ad4b84929 admin admin] PATH_INFO: `/v3/users/a649b40eac36441bbcadb219a21a8907` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:50.112228 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b461196-2b6a-4bc4-b593-af4ad4b84929 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a649b40eac36441bbcadb219a21a8907)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:50.114187 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b461196-2b6a-4bc4-b593-af4ad4b84929 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:50.187687 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9b461196-2b6a-4bc4-b593-af4ad4b84929 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a649b40eac36441bbcadb219a21a8907'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:50.201630 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9b461196-2b6a-4bc4-b593-af4ad4b84929 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a649b40eac36441bbcadb219a21a8907'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:50.202357 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9b461196-2b6a-4bc4-b593-af4ad4b84929 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a649b40eac36441bbcadb219a21a8907'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:50.204930 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9b461196-2b6a-4bc4-b593-af4ad4b84929 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a649b40eac36441bbcadb219a21a8907'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:50.216165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9b461196-2b6a-4bc4-b593-af4ad4b84929 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a649b40eac36441bbcadb219a21a8907'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:50.261608 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 917/1845] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:57:49 2022] DELETE /identity/v3/users/a649b40eac36441bbcadb219a21a8907 => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:57:50.266609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1400871d-7a1f-48f5-8731-b713e01334c1 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:50.267469 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1400871d-7a1f-48f5-8731-b713e01334c1 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:50.267872 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1400871d-7a1f-48f5-8731-b713e01334c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:50.298723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-1400871d-7a1f-48f5-8731-b713e01334c1 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:50.300040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-1400871d-7a1f-48f5-8731-b713e01334c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:50.443241 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 929/1846] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:50.451599 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b461196-2b6a-4bc4-b593-af4ad4b84929 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:50.455956 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-9b461196-2b6a-4bc4-b593-af4ad4b84929 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:50.590375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b461196-2b6a-4bc4-b593-af4ad4b84929 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:50.595373 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e802ce-d471-4af0-b9bb-0987e52717ef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:50.596389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e802ce-d471-4af0-b9bb-0987e52717ef service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:50.596798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e802ce-d471-4af0-b9bb-0987e52717ef service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:50.597307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e802ce-d471-4af0-b9bb-0987e52717ef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:50.612795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e802ce-d471-4af0-b9bb-0987e52717ef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:50.615816 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e802ce-d471-4af0-b9bb-0987e52717ef service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:50.629923 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 918/1847] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:51.002639 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1400871d-7a1f-48f5-8731-b713e01334c1 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:51.004562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-1400871d-7a1f-48f5-8731-b713e01334c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:51.033654 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1400871d-7a1f-48f5-8731-b713e01334c1 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:51.045643 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0d0b4b-1bcd-4d14-97c7-28d7972759bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:51.046316 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0d0b4b-1bcd-4d14-97c7-28d7972759bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:51.046617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0d0b4b-1bcd-4d14-97c7-28d7972759bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:51.046892 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0d0b4b-1bcd-4d14-97c7-28d7972759bf admin admin] PATH_INFO: `/v3/projects/81b774e3b54b4fe3bf25502b2ec64484` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:51.048045 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0d0b4b-1bcd-4d14-97c7-28d7972759bf admin admin] RBAC: Authorizing `identity:delete_project(project_id=81b774e3b54b4fe3bf25502b2ec64484)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:51.050351 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0d0b4b-1bcd-4d14-97c7-28d7972759bf admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:51.109828 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-0b0d0b4b-1bcd-4d14-97c7-28d7972759bf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '81b774e3b54b4fe3bf25502b2ec64484', 'request_id': 'req-0b0d0b4b-1bcd-4d14-97c7-28d7972759bf'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:51.117305 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-0b0d0b4b-1bcd-4d14-97c7-28d7972759bf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '81b774e3b54b4fe3bf25502b2ec64484', 'request_id': 'req-0b0d0b4b-1bcd-4d14-97c7-28d7972759bf'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:51.126633 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-0b0d0b4b-1bcd-4d14-97c7-28d7972759bf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '81b774e3b54b4fe3bf25502b2ec64484', 'request_id': 'req-0b0d0b4b-1bcd-4d14-97c7-28d7972759bf'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:51.140648 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-0b0d0b4b-1bcd-4d14-97c7-28d7972759bf admin admin] The token cache is being invalidate because project 81b774e3b54b4fe3bf25502b2ec64484 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:57:51.140992 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-0b0d0b4b-1bcd-4d14-97c7-28d7972759bf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:51.141769 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-0b0d0b4b-1bcd-4d14-97c7-28d7972759bf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:51.143515 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 930/1848] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:57:51 2022] DELETE /identity/v3/projects/81b774e3b54b4fe3bf25502b2ec64484 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:51.149484 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f09b28-d892-412a-9e64-82817cff773c None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:51.150047 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f09b28-d892-412a-9e64-82817cff773c None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:51.150047 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f09b28-d892-412a-9e64-82817cff773c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:51.174382 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-f0f09b28-d892-412a-9e64-82817cff773c None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:51.175157 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-f0f09b28-d892-412a-9e64-82817cff773c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:51.194816 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:51.260479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 919/1849] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:51.276656 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b094fe2b-47b0-4de3-b69e-e3e49e055ffc None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:51.277208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b094fe2b-47b0-4de3-b69e-e3e49e055ffc None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:51.277344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b094fe2b-47b0-4de3-b69e-e3e49e055ffc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:51.299494 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-b094fe2b-47b0-4de3-b69e-e3e49e055ffc None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:51.300821 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-b094fe2b-47b0-4de3-b69e-e3e49e055ffc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:51.309594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 931/1850] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:51.314986 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f09b28-d892-412a-9e64-82817cff773c None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:51.328024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f09b28-d892-412a-9e64-82817cff773c None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:51.335162 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f4cb62-3e63-4f90-9303-3401e3023447 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:51.335923 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f4cb62-3e63-4f90-9303-3401e3023447 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:51.336558 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f4cb62-3e63-4f90-9303-3401e3023447 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:51.336679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f4cb62-3e63-4f90-9303-3401e3023447 admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:51.337514 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f4cb62-3e63-4f90-9303-3401e3023447 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:57:51.337885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f4cb62-3e63-4f90-9303-3401e3023447 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:51.340162 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f4cb62-3e63-4f90-9303-3401e3023447 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:51.359081 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 920/1851] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:57:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:57:51.364026 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b094fe2b-47b0-4de3-b69e-e3e49e055ffc None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:51.377144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b094fe2b-47b0-4de3-b69e-e3e49e055ffc None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:51.382755 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da3d32d-391a-47e5-aeb6-a25227066bb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:51.383789 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da3d32d-391a-47e5-aeb6-a25227066bb9 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:51.384075 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da3d32d-391a-47e5-aeb6-a25227066bb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:51.384349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da3d32d-391a-47e5-aeb6-a25227066bb9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:51.385028 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da3d32d-391a-47e5-aeb6-a25227066bb9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:51.386989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da3d32d-391a-47e5-aeb6-a25227066bb9 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:51.418658 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 932/1852] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:57:51 2022] POST /identity/v3/projects => generated 373 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:57:51.423345 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f4cb62-3e63-4f90-9303-3401e3023447 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:51.436002 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f4cb62-3e63-4f90-9303-3401e3023447 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:51.564859 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29afd30f-6602-4dbc-9d93-f211b16bfb0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:51.565669 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29afd30f-6602-4dbc-9d93-f211b16bfb0e admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:51.565989 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29afd30f-6602-4dbc-9d93-f211b16bfb0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:51.566300 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29afd30f-6602-4dbc-9d93-f211b16bfb0e admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:51.569424 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29afd30f-6602-4dbc-9d93-f211b16bfb0e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:51.573056 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29afd30f-6602-4dbc-9d93-f211b16bfb0e admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:51.611131 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 921/1853] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:57:51 2022] POST /identity/v3/users => generated 287 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:57:51.611867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da3d32d-391a-47e5-aeb6-a25227066bb9 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:51.626054 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da3d32d-391a-47e5-aeb6-a25227066bb9 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:51.642300 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef980cd0-536e-4cad-a06e-b71ba1b2f727 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:51.643181 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef980cd0-536e-4cad-a06e-b71ba1b2f727 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:51.643524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef980cd0-536e-4cad-a06e-b71ba1b2f727 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:51.643915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef980cd0-536e-4cad-a06e-b71ba1b2f727 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:51.644689 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef980cd0-536e-4cad-a06e-b71ba1b2f727 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:51.647305 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef980cd0-536e-4cad-a06e-b71ba1b2f727 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:51.667977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29afd30f-6602-4dbc-9d93-f211b16bfb0e admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:51.669245 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 933/1854] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:51 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:57:51.699922 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29afd30f-6602-4dbc-9d93-f211b16bfb0e admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:51.710368 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7276b80d-d06f-4c82-b58f-944c5debf223 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:51.711398 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7276b80d-d06f-4c82-b58f-944c5debf223 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:51.711525 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7276b80d-d06f-4c82-b58f-944c5debf223 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:51.711779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7276b80d-d06f-4c82-b58f-944c5debf223 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:51.713647 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7276b80d-d06f-4c82-b58f-944c5debf223 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:51.716053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7276b80d-d06f-4c82-b58f-944c5debf223 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:51.749287 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 922/1855] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:51 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:57:51.757554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef980cd0-536e-4cad-a06e-b71ba1b2f727 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:51.773807 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef980cd0-536e-4cad-a06e-b71ba1b2f727 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:51.783636 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111ca0ce-715e-4946-bae0-245c178f32c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:51.784693 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111ca0ce-715e-4946-bae0-245c178f32c8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:51.785288 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111ca0ce-715e-4946-bae0-245c178f32c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:51.785973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111ca0ce-715e-4946-bae0-245c178f32c8 admin admin] PATH_INFO: `/v3/projects/2bdaf54eb5704437a1230a9b8c2a8dad/users/0380fe7fa37a411ba02fcca3e29eb670/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:51.816772 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111ca0ce-715e-4946-bae0-245c178f32c8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2bdaf54eb5704437a1230a9b8c2a8dad, user_id=0380fe7fa37a411ba02fcca3e29eb670, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:51.820005 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111ca0ce-715e-4946-bae0-245c178f32c8 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:51.847379 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 934/1856] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:57:51 2022] PUT /identity/v3/projects/2bdaf54eb5704437a1230a9b8c2a8dad/users/0380fe7fa37a411ba02fcca3e29eb670/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:51.863666 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0e3130-a0c2-4abb-8e91-f5246dcd24b6 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:51.863953 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0e3130-a0c2-4abb-8e91-f5246dcd24b6 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:51.864231 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0e3130-a0c2-4abb-8e91-f5246dcd24b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:51.944811 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-4c0e3130-a0c2-4abb-8e91-f5246dcd24b6 None None] MFA Rules not processed for user `0380fe7fa37a411ba02fcca3e29eb670`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:51.946078 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-4c0e3130-a0c2-4abb-8e91-f5246dcd24b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:52.082570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 923/1857] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:51 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:52.088614 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111ca0ce-715e-4946-bae0-245c178f32c8 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:52.126179 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:52.193353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111ca0ce-715e-4946-bae0-245c178f32c8 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:52.201779 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b83ab79-7362-4cba-a0ca-f93d81aa2352 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:52.202532 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b83ab79-7362-4cba-a0ca-f93d81aa2352 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:52.202859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b83ab79-7362-4cba-a0ca-f93d81aa2352 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:52.203138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b83ab79-7362-4cba-a0ca-f93d81aa2352 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:52.203812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b83ab79-7362-4cba-a0ca-f93d81aa2352 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:52.206676 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b83ab79-7362-4cba-a0ca-f93d81aa2352 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:52.258739 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 935/1858] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:57:52 2022] POST /identity/v3/projects => generated 375 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:57:52.263315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0e3130-a0c2-4abb-8e91-f5246dcd24b6 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:52.281368 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0e3130-a0c2-4abb-8e91-f5246dcd24b6 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:52.321032 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:52.424314 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc1b889-1964-417c-9a20-fa2cf3f6a0b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:52.424652 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc1b889-1964-417c-9a20-fa2cf3f6a0b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:52.424962 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc1b889-1964-417c-9a20-fa2cf3f6a0b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:52.425302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc1b889-1964-417c-9a20-fa2cf3f6a0b0 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:52.426345 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc1b889-1964-417c-9a20-fa2cf3f6a0b0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:52.428666 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc1b889-1964-417c-9a20-fa2cf3f6a0b0 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:52.469105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 924/1859] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:57:52 2022] POST /identity/v3/users => generated 294 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:57:52.473464 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b83ab79-7362-4cba-a0ca-f93d81aa2352 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:52.490783 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b83ab79-7362-4cba-a0ca-f93d81aa2352 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:52.501001 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dc7f9f-a18c-4f78-914e-7687977e5404 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:52.502933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dc7f9f-a18c-4f78-914e-7687977e5404 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:52.504622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dc7f9f-a18c-4f78-914e-7687977e5404 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:52.506187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dc7f9f-a18c-4f78-914e-7687977e5404 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:52.509478 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24dc7f9f-a18c-4f78-914e-7687977e5404 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:52.512138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24dc7f9f-a18c-4f78-914e-7687977e5404 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:52.531441 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 936/1860] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:52 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:57:52.535494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc1b889-1964-417c-9a20-fa2cf3f6a0b0 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:52.547176 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc1b889-1964-417c-9a20-fa2cf3f6a0b0 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:52.553109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f314e5-225d-49be-aa1a-1059203d5679 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:52.553814 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f314e5-225d-49be-aa1a-1059203d5679 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:52.554086 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f314e5-225d-49be-aa1a-1059203d5679 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:52.554354 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f314e5-225d-49be-aa1a-1059203d5679 admin admin] PATH_INFO: `/v3/projects/7c8642aa1c9447489508b36a36ba0851/users/4d87522540fb4eab89b5eaf12a110969/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:52.571416 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f314e5-225d-49be-aa1a-1059203d5679 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7c8642aa1c9447489508b36a36ba0851, user_id=4d87522540fb4eab89b5eaf12a110969, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:52.574367 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f314e5-225d-49be-aa1a-1059203d5679 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:52.592628 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 925/1861] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:57:52 2022] PUT /identity/v3/projects/7c8642aa1c9447489508b36a36ba0851/users/4d87522540fb4eab89b5eaf12a110969/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:52.596916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dc7f9f-a18c-4f78-914e-7687977e5404 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:52.609548 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dc7f9f-a18c-4f78-914e-7687977e5404 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:52.703146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6b10f2-f9b3-48c4-a389-96fbdb07ccc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:52.703963 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6b10f2-f9b3-48c4-a389-96fbdb07ccc8 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:52.704355 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6b10f2-f9b3-48c4-a389-96fbdb07ccc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:52.704768 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6b10f2-f9b3-48c4-a389-96fbdb07ccc8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:52.705529 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6b10f2-f9b3-48c4-a389-96fbdb07ccc8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:52.708262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6b10f2-f9b3-48c4-a389-96fbdb07ccc8 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:52.724185 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 937/1862] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:57:52 2022] GET /identity/v3/roles => generated 1420 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:57:52.729778 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f314e5-225d-49be-aa1a-1059203d5679 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:52.743293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f314e5-225d-49be-aa1a-1059203d5679 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:52.750101 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea5d2ed-d8e2-4f31-b529-e5788ba9e1fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:52.751255 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea5d2ed-d8e2-4f31-b529-e5788ba9e1fa admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:52.751749 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea5d2ed-d8e2-4f31-b529-e5788ba9e1fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:52.752229 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea5d2ed-d8e2-4f31-b529-e5788ba9e1fa admin admin] PATH_INFO: `/v3/projects/7c8642aa1c9447489508b36a36ba0851/users/4d87522540fb4eab89b5eaf12a110969/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:52.755528 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea5d2ed-d8e2-4f31-b529-e5788ba9e1fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=7c8642aa1c9447489508b36a36ba0851, user_id=4d87522540fb4eab89b5eaf12a110969, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:52.758495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea5d2ed-d8e2-4f31-b529-e5788ba9e1fa admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:52.779368 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 926/1863] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:57:52 2022] PUT /identity/v3/projects/7c8642aa1c9447489508b36a36ba0851/users/4d87522540fb4eab89b5eaf12a110969/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:52.792710 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5c8036-4839-4717-a12c-9da522678af5 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:52.793149 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5c8036-4839-4717-a12c-9da522678af5 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:52.793586 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5c8036-4839-4717-a12c-9da522678af5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:52.831775 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-8b5c8036-4839-4717-a12c-9da522678af5 None None] MFA Rules not processed for user `4d87522540fb4eab89b5eaf12a110969`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:52.833149 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-8b5c8036-4839-4717-a12c-9da522678af5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:52.930588 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 938/1864] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:52 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:52.947933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea5d2ed-d8e2-4f31-b529-e5788ba9e1fa admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:52.950995 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-bea5d2ed-d8e2-4f31-b529-e5788ba9e1fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:53.037217 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea5d2ed-d8e2-4f31-b529-e5788ba9e1fa admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:53.045224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be969d31-cdf5-4e34-9b10-8c1029fafc81 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:53.045957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be969d31-cdf5-4e34-9b10-8c1029fafc81 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:53.046241 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be969d31-cdf5-4e34-9b10-8c1029fafc81 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:53.046556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be969d31-cdf5-4e34-9b10-8c1029fafc81 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:53.163308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be969d31-cdf5-4e34-9b10-8c1029fafc81 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:53.165526 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be969d31-cdf5-4e34-9b10-8c1029fafc81 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:53.177672 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 927/1865] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:52 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 231 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:53.657187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab295e0-e2d9-444b-b66d-9522c31aed23 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:53.658022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab295e0-e2d9-444b-b66d-9522c31aed23 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:53.659831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab295e0-e2d9-444b-b66d-9522c31aed23 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:53.662676 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 939/1866] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 18 00:57:53 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:57:53.684610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598b2bcb-da06-4a4f-872e-d440041c15f4 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:53.685234 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598b2bcb-da06-4a4f-872e-d440041c15f4 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:53.685623 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598b2bcb-da06-4a4f-872e-d440041c15f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:53.757085 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-598b2bcb-da06-4a4f-872e-d440041c15f4 None None] MFA Rules not processed for user `a59d73fd45a54651930d52300704fdb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:53.758730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-598b2bcb-da06-4a4f-872e-d440041c15f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:53.784607 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:53.907760 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 928/1867] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 18 00:57:53 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:54.215223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab295e0-e2d9-444b-b66d-9522c31aed23 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:54.217269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3ab295e0-e2d9-444b-b66d-9522c31aed23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:54.266720 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab295e0-e2d9-444b-b66d-9522c31aed23 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:54.286278 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34822515-732a-491c-a685-cdacff9f91a7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:54.287718 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34822515-732a-491c-a685-cdacff9f91a7 service nova] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:54.288145 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34822515-732a-491c-a685-cdacff9f91a7 service nova] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:54.288569 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34822515-732a-491c-a685-cdacff9f91a7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:54.302155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34822515-732a-491c-a685-cdacff9f91a7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:54.304125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34822515-732a-491c-a685-cdacff9f91a7 service nova] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:54.318652 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 940/1868] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:54 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:58.545517 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598b2bcb-da06-4a4f-872e-d440041c15f4 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:58.613938 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598b2bcb-da06-4a4f-872e-d440041c15f4 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:58.622686 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3baf9a-33e8-47c0-80cc-5c3c9fcccbae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:58.623362 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3baf9a-33e8-47c0-80cc-5c3c9fcccbae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:58.623700 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3baf9a-33e8-47c0-80cc-5c3c9fcccbae admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:58.624298 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3baf9a-33e8-47c0-80cc-5c3c9fcccbae admin admin] PATH_INFO: `/v3/users/0380fe7fa37a411ba02fcca3e29eb670` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:58.625529 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3baf9a-33e8-47c0-80cc-5c3c9fcccbae admin admin] RBAC: Authorizing `identity:delete_user(user_id=0380fe7fa37a411ba02fcca3e29eb670)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:58.627939 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3baf9a-33e8-47c0-80cc-5c3c9fcccbae admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:58.690815 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0d3baf9a-33e8-47c0-80cc-5c3c9fcccbae admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0380fe7fa37a411ba02fcca3e29eb670'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:58.700648 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0d3baf9a-33e8-47c0-80cc-5c3c9fcccbae admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0380fe7fa37a411ba02fcca3e29eb670'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:58.701571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0d3baf9a-33e8-47c0-80cc-5c3c9fcccbae admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0380fe7fa37a411ba02fcca3e29eb670'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:58.702299 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0d3baf9a-33e8-47c0-80cc-5c3c9fcccbae admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0380fe7fa37a411ba02fcca3e29eb670'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:58.711384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0d3baf9a-33e8-47c0-80cc-5c3c9fcccbae admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0380fe7fa37a411ba02fcca3e29eb670'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:58.730307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 929/1869] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:57:58 2022] DELETE /identity/v3/users/0380fe7fa37a411ba02fcca3e29eb670 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:57:58.735157 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247adbeb-b859-468f-a415-ead6356aa970 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:58.735633 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247adbeb-b859-468f-a415-ead6356aa970 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:58.735893 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247adbeb-b859-468f-a415-ead6356aa970 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:58.744467 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:58.760842 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-247adbeb-b859-468f-a415-ead6356aa970 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:58.762186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-247adbeb-b859-468f-a415-ead6356aa970 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:58.809615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3baf9a-33e8-47c0-80cc-5c3c9fcccbae admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:58.811627 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-0d3baf9a-33e8-47c0-80cc-5c3c9fcccbae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:58.829711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 941/1870] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:58.843067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247adbeb-b859-468f-a415-ead6356aa970 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:58.844545 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-247adbeb-b859-468f-a415-ead6356aa970 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:58.861362 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:58.889766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3baf9a-33e8-47c0-80cc-5c3c9fcccbae admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:58.895445 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81f9df4-56ec-4440-9b13-bfc813f32ae8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:58.896149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81f9df4-56ec-4440-9b13-bfc813f32ae8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:58.896430 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81f9df4-56ec-4440-9b13-bfc813f32ae8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:58.896724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81f9df4-56ec-4440-9b13-bfc813f32ae8 admin admin] PATH_INFO: `/v3/users/51a2ade99d324a07a6e350ee8ef28a77` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:58.897869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81f9df4-56ec-4440-9b13-bfc813f32ae8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=51a2ade99d324a07a6e350ee8ef28a77)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:58.899889 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81f9df4-56ec-4440-9b13-bfc813f32ae8 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:58.909426 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247adbeb-b859-468f-a415-ead6356aa970 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:58.914087 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1563a1c2-82f1-4f58-b1fa-b1e8a8fd0932 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:58.914732 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1563a1c2-82f1-4f58-b1fa-b1e8a8fd0932 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:58.915019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1563a1c2-82f1-4f58-b1fa-b1e8a8fd0932 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:58.915286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1563a1c2-82f1-4f58-b1fa-b1e8a8fd0932 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:58.925440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1563a1c2-82f1-4f58-b1fa-b1e8a8fd0932 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:58.927393 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1563a1c2-82f1-4f58-b1fa-b1e8a8fd0932 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:58.942052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 942/1871] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:57:58.954170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b81f9df4-56ec-4440-9b13-bfc813f32ae8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '51a2ade99d324a07a6e350ee8ef28a77'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:58.964772 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b81f9df4-56ec-4440-9b13-bfc813f32ae8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '51a2ade99d324a07a6e350ee8ef28a77'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:58.965671 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b81f9df4-56ec-4440-9b13-bfc813f32ae8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '51a2ade99d324a07a6e350ee8ef28a77'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:58.966547 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b81f9df4-56ec-4440-9b13-bfc813f32ae8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '51a2ade99d324a07a6e350ee8ef28a77'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:58.975209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b81f9df4-56ec-4440-9b13-bfc813f32ae8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '51a2ade99d324a07a6e350ee8ef28a77'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:59.001762 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 930/1872] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:57:58 2022] DELETE /identity/v3/users/51a2ade99d324a07a6e350ee8ef28a77 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:57:59.009397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8ffc06-6cb7-41df-ac5a-073ded552007 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:59.009765 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8ffc06-6cb7-41df-ac5a-073ded552007 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:59.010254 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8ffc06-6cb7-41df-ac5a-073ded552007 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:59.051548 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-fa8ffc06-6cb7-41df-ac5a-073ded552007 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:57:59.054339 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-fa8ffc06-6cb7-41df-ac5a-073ded552007 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:59.160833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 943/1873] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:57:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:57:59.182523 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81f9df4-56ec-4440-9b13-bfc813f32ae8 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:59.183988 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-b81f9df4-56ec-4440-9b13-bfc813f32ae8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:59.279282 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81f9df4-56ec-4440-9b13-bfc813f32ae8 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:59.289197 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953e16de-17e0-4e16-a6ce-9eeb5ae95385 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:59.289875 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953e16de-17e0-4e16-a6ce-9eeb5ae95385 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:59.290345 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953e16de-17e0-4e16-a6ce-9eeb5ae95385 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:59.290641 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953e16de-17e0-4e16-a6ce-9eeb5ae95385 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:59.296621 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8ffc06-6cb7-41df-ac5a-073ded552007 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:59.300384 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-fa8ffc06-6cb7-41df-ac5a-073ded552007 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:57:59.332903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8ffc06-6cb7-41df-ac5a-073ded552007 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:59.341778 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953e16de-17e0-4e16-a6ce-9eeb5ae95385 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:59.344507 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953e16de-17e0-4e16-a6ce-9eeb5ae95385 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:59.346821 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f311779-15ee-49ed-a7ad-b3c738b2b474 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:59.347491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f311779-15ee-49ed-a7ad-b3c738b2b474 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:59.347774 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f311779-15ee-49ed-a7ad-b3c738b2b474 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:59.348050 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f311779-15ee-49ed-a7ad-b3c738b2b474 admin admin] PATH_INFO: `/v3/projects/2bdaf54eb5704437a1230a9b8c2a8dad` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:59.349900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f311779-15ee-49ed-a7ad-b3c738b2b474 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2bdaf54eb5704437a1230a9b8c2a8dad)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:59.351919 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f311779-15ee-49ed-a7ad-b3c738b2b474 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:59.393584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 931/1874] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:57:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 211 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:57:59.475561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:59.485070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7f311779-15ee-49ed-a7ad-b3c738b2b474 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2bdaf54eb5704437a1230a9b8c2a8dad', 'request_id': 'req-7f311779-15ee-49ed-a7ad-b3c738b2b474'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:59.494254 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7f311779-15ee-49ed-a7ad-b3c738b2b474 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2bdaf54eb5704437a1230a9b8c2a8dad', 'request_id': 'req-7f311779-15ee-49ed-a7ad-b3c738b2b474'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:59.506626 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7f311779-15ee-49ed-a7ad-b3c738b2b474 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2bdaf54eb5704437a1230a9b8c2a8dad', 'request_id': 'req-7f311779-15ee-49ed-a7ad-b3c738b2b474'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:59.542855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7f311779-15ee-49ed-a7ad-b3c738b2b474 admin admin] The token cache is being invalidate because project 2bdaf54eb5704437a1230a9b8c2a8dad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:57:59.543673 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7f311779-15ee-49ed-a7ad-b3c738b2b474 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:59.544908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7f311779-15ee-49ed-a7ad-b3c738b2b474 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:59.548105 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 944/1875] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:57:59 2022] DELETE /identity/v3/projects/2bdaf54eb5704437a1230a9b8c2a8dad => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:57:59.553262 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953e16de-17e0-4e16-a6ce-9eeb5ae95385 service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:57:59.583236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953e16de-17e0-4e16-a6ce-9eeb5ae95385 service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:57:59.700961 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:57:59.710636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82878327-b862-4a50-8b82-bc01bde2898a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:57:59.711869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82878327-b862-4a50-8b82-bc01bde2898a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:57:59.712665 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82878327-b862-4a50-8b82-bc01bde2898a admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:57:59.714206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82878327-b862-4a50-8b82-bc01bde2898a admin admin] PATH_INFO: `/v3/users/4d87522540fb4eab89b5eaf12a110969` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:57:59.716155 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82878327-b862-4a50-8b82-bc01bde2898a admin admin] RBAC: Authorizing `identity:delete_user(user_id=4d87522540fb4eab89b5eaf12a110969)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:57:59.718779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82878327-b862-4a50-8b82-bc01bde2898a admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:57:59.787038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-82878327-b862-4a50-8b82-bc01bde2898a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4d87522540fb4eab89b5eaf12a110969'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:59.795997 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-82878327-b862-4a50-8b82-bc01bde2898a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4d87522540fb4eab89b5eaf12a110969'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:59.796991 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-82878327-b862-4a50-8b82-bc01bde2898a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4d87522540fb4eab89b5eaf12a110969'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:59.797828 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-82878327-b862-4a50-8b82-bc01bde2898a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4d87522540fb4eab89b5eaf12a110969'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:59.810605 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-82878327-b862-4a50-8b82-bc01bde2898a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4d87522540fb4eab89b5eaf12a110969'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:57:59.844516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 932/1876] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:57:59 2022] DELETE /identity/v3/users/4d87522540fb4eab89b5eaf12a110969 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:00.298164 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f311779-15ee-49ed-a7ad-b3c738b2b474 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:00.299677 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-7f311779-15ee-49ed-a7ad-b3c738b2b474 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:00.317019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82878327-b862-4a50-8b82-bc01bde2898a admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:00.325162 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-82878327-b862-4a50-8b82-bc01bde2898a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:00.338123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f311779-15ee-49ed-a7ad-b3c738b2b474 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:00.423085 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82878327-b862-4a50-8b82-bc01bde2898a admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:00.423473 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fdbd64-5cb6-4bdf-80fa-bd4e78fa0f7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:00.424215 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fdbd64-5cb6-4bdf-80fa-bd4e78fa0f7b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:00.424568 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fdbd64-5cb6-4bdf-80fa-bd4e78fa0f7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:00.424917 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fdbd64-5cb6-4bdf-80fa-bd4e78fa0f7b admin admin] PATH_INFO: `/v3/projects/7c8642aa1c9447489508b36a36ba0851` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:00.426162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fdbd64-5cb6-4bdf-80fa-bd4e78fa0f7b admin admin] RBAC: Authorizing `identity:delete_project(project_id=7c8642aa1c9447489508b36a36ba0851)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:00.428313 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fdbd64-5cb6-4bdf-80fa-bd4e78fa0f7b admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:00.428553 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398a4586-e49d-4f71-9394-1968b30c8699 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:00.429222 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398a4586-e49d-4f71-9394-1968b30c8699 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:00.429577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398a4586-e49d-4f71-9394-1968b30c8699 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:00.429858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398a4586-e49d-4f71-9394-1968b30c8699 admin admin] PATH_INFO: `/v3/projects/ee5ad84a77b74cf9ab2d9bc7959fb838` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:00.431350 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398a4586-e49d-4f71-9394-1968b30c8699 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ee5ad84a77b74cf9ab2d9bc7959fb838)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:00.433298 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398a4586-e49d-4f71-9394-1968b30c8699 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:00.494476 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-59fdbd64-5cb6-4bdf-80fa-bd4e78fa0f7b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7c8642aa1c9447489508b36a36ba0851', 'request_id': 'req-59fdbd64-5cb6-4bdf-80fa-bd4e78fa0f7b'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:00.503659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-59fdbd64-5cb6-4bdf-80fa-bd4e78fa0f7b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7c8642aa1c9447489508b36a36ba0851', 'request_id': 'req-59fdbd64-5cb6-4bdf-80fa-bd4e78fa0f7b'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:00.516391 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-59fdbd64-5cb6-4bdf-80fa-bd4e78fa0f7b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7c8642aa1c9447489508b36a36ba0851', 'request_id': 'req-59fdbd64-5cb6-4bdf-80fa-bd4e78fa0f7b'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:00.517759 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-398a4586-e49d-4f71-9394-1968b30c8699 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ee5ad84a77b74cf9ab2d9bc7959fb838', 'request_id': 'req-398a4586-e49d-4f71-9394-1968b30c8699'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:00.531379 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-398a4586-e49d-4f71-9394-1968b30c8699 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ee5ad84a77b74cf9ab2d9bc7959fb838', 'request_id': 'req-398a4586-e49d-4f71-9394-1968b30c8699'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:00.534805 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-59fdbd64-5cb6-4bdf-80fa-bd4e78fa0f7b admin admin] The token cache is being invalidate because project 7c8642aa1c9447489508b36a36ba0851 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:00.535112 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-59fdbd64-5cb6-4bdf-80fa-bd4e78fa0f7b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:00.536943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-59fdbd64-5cb6-4bdf-80fa-bd4e78fa0f7b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:00.539029 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 945/1877] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:00 2022] DELETE /identity/v3/projects/7c8642aa1c9447489508b36a36ba0851 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:00.544507 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d0ee29-4f0c-4053-b068-70440737cddb None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:00.544793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d0ee29-4f0c-4053-b068-70440737cddb None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:00.545071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d0ee29-4f0c-4053-b068-70440737cddb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:00.549275 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-398a4586-e49d-4f71-9394-1968b30c8699 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ee5ad84a77b74cf9ab2d9bc7959fb838', 'request_id': 'req-398a4586-e49d-4f71-9394-1968b30c8699'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:00.565314 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-398a4586-e49d-4f71-9394-1968b30c8699 admin admin] The token cache is being invalidate because project ee5ad84a77b74cf9ab2d9bc7959fb838 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:00.565999 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-398a4586-e49d-4f71-9394-1968b30c8699 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:00.566782 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-398a4586-e49d-4f71-9394-1968b30c8699 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:00.568804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-44d0ee29-4f0c-4053-b068-70440737cddb None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:00.570130 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-44d0ee29-4f0c-4053-b068-70440737cddb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:00.573487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 933/1878] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:00 2022] DELETE /identity/v3/projects/ee5ad84a77b74cf9ab2d9bc7959fb838 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:00.574983 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398a4586-e49d-4f71-9394-1968b30c8699 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:00.576453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-398a4586-e49d-4f71-9394-1968b30c8699 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:00.589281 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:00.657660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:00.674992 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 946/1879] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:00.684270 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9a9bb1-9eec-439d-b131-75720e5053bf None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:00.684604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9a9bb1-9eec-439d-b131-75720e5053bf None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:00.684883 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9a9bb1-9eec-439d-b131-75720e5053bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:00.694902 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398a4586-e49d-4f71-9394-1968b30c8699 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:00.701986 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521e4033-ff33-4bf8-8555-c23f46d12664 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:00.703025 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521e4033-ff33-4bf8-8555-c23f46d12664 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:00.703341 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521e4033-ff33-4bf8-8555-c23f46d12664 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:00.703797 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521e4033-ff33-4bf8-8555-c23f46d12664 admin admin] PATH_INFO: `/v3/users/898c445d3f1d40fc945fa70899aaf3b9` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:00.705912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521e4033-ff33-4bf8-8555-c23f46d12664 admin admin] RBAC: Authorizing `identity:delete_user(user_id=898c445d3f1d40fc945fa70899aaf3b9)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:00.709276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521e4033-ff33-4bf8-8555-c23f46d12664 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:00.717040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-0c9a9bb1-9eec-439d-b131-75720e5053bf None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:00.718705 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-0c9a9bb1-9eec-439d-b131-75720e5053bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:00.733078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 947/1880] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:00.734377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9a9bb1-9eec-439d-b131-75720e5053bf None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:00.749945 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9a9bb1-9eec-439d-b131-75720e5053bf None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:00.754799 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192c0e60-6112-43d8-8616-e1189eeee3f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:00.755497 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192c0e60-6112-43d8-8616-e1189eeee3f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:00.756281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192c0e60-6112-43d8-8616-e1189eeee3f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:00.756556 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192c0e60-6112-43d8-8616-e1189eeee3f7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:00.757217 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192c0e60-6112-43d8-8616-e1189eeee3f7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:58:00.757520 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192c0e60-6112-43d8-8616-e1189eeee3f7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:00.759183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192c0e60-6112-43d8-8616-e1189eeee3f7 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:00.772569 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 948/1881] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:58:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:58:00.776975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192c0e60-6112-43d8-8616-e1189eeee3f7 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:00.788790 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192c0e60-6112-43d8-8616-e1189eeee3f7 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:00.795622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591fe7cd-aac2-4c4f-9a29-67089a1637bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:00.796398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591fe7cd-aac2-4c4f-9a29-67089a1637bf admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:00.796522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-521e4033-ff33-4bf8-8555-c23f46d12664 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '898c445d3f1d40fc945fa70899aaf3b9'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:00.796680 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591fe7cd-aac2-4c4f-9a29-67089a1637bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:00.797039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591fe7cd-aac2-4c4f-9a29-67089a1637bf admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:00.797686 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591fe7cd-aac2-4c4f-9a29-67089a1637bf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:00.800809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591fe7cd-aac2-4c4f-9a29-67089a1637bf admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:00.806735 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-521e4033-ff33-4bf8-8555-c23f46d12664 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '898c445d3f1d40fc945fa70899aaf3b9'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:00.808239 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-521e4033-ff33-4bf8-8555-c23f46d12664 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '898c445d3f1d40fc945fa70899aaf3b9'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:00.809037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-521e4033-ff33-4bf8-8555-c23f46d12664 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '898c445d3f1d40fc945fa70899aaf3b9'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:00.825173 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-521e4033-ff33-4bf8-8555-c23f46d12664 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '898c445d3f1d40fc945fa70899aaf3b9'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:00.848000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 949/1882] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:00 2022] POST /identity/v3/projects => generated 371 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:00.850224 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591fe7cd-aac2-4c4f-9a29-67089a1637bf admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:00.852724 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-591fe7cd-aac2-4c4f-9a29-67089a1637bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:00.873327 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 934/1883] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:00 2022] DELETE /identity/v3/users/898c445d3f1d40fc945fa70899aaf3b9 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:01.021295 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591fe7cd-aac2-4c4f-9a29-67089a1637bf admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:01.109048 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc54c18f-cc29-41e3-9d30-ccff08b8f4a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:01.109731 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc54c18f-cc29-41e3-9d30-ccff08b8f4a0 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:01.110012 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc54c18f-cc29-41e3-9d30-ccff08b8f4a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:01.110340 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc54c18f-cc29-41e3-9d30-ccff08b8f4a0 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:01.111117 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc54c18f-cc29-41e3-9d30-ccff08b8f4a0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:01.113487 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc54c18f-cc29-41e3-9d30-ccff08b8f4a0 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:01.147908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 950/1884] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:00 2022] POST /identity/v3/users => generated 286 bytes in 299 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:01.152932 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521e4033-ff33-4bf8-8555-c23f46d12664 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:01.169944 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521e4033-ff33-4bf8-8555-c23f46d12664 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:01.175506 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97dd7db-bc3e-43e8-95c4-a30a4310eca1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:01.176266 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97dd7db-bc3e-43e8-95c4-a30a4310eca1 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:01.176622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97dd7db-bc3e-43e8-95c4-a30a4310eca1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:01.176973 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97dd7db-bc3e-43e8-95c4-a30a4310eca1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:01.177734 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97dd7db-bc3e-43e8-95c4-a30a4310eca1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:01.180037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97dd7db-bc3e-43e8-95c4-a30a4310eca1 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:01.195380 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 935/1885] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:01 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:01.199292 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc54c18f-cc29-41e3-9d30-ccff08b8f4a0 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:01.216395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc54c18f-cc29-41e3-9d30-ccff08b8f4a0 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:01.222052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490ccbdb-a17f-4ece-9cbd-eaa58aeda287 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:01.222936 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490ccbdb-a17f-4ece-9cbd-eaa58aeda287 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:01.223443 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490ccbdb-a17f-4ece-9cbd-eaa58aeda287 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:01.223895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490ccbdb-a17f-4ece-9cbd-eaa58aeda287 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:01.225679 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490ccbdb-a17f-4ece-9cbd-eaa58aeda287 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:01.227585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490ccbdb-a17f-4ece-9cbd-eaa58aeda287 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:01.241892 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 951/1886] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:01 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:01.245631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97dd7db-bc3e-43e8-95c4-a30a4310eca1 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:01.260465 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97dd7db-bc3e-43e8-95c4-a30a4310eca1 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:01.271493 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744fbf01-701d-4a5e-aced-cb3ea4fabfde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:01.272360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744fbf01-701d-4a5e-aced-cb3ea4fabfde admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:01.272614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744fbf01-701d-4a5e-aced-cb3ea4fabfde admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:01.272953 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744fbf01-701d-4a5e-aced-cb3ea4fabfde admin admin] PATH_INFO: `/v3/projects/c107c89986ca4ec7adb56f731a567459/users/5bba915f4a374bba9c8f934b3f742b6c/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:01.293897 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744fbf01-701d-4a5e-aced-cb3ea4fabfde admin admin] RBAC: Authorizing `identity:create_grant(project_id=c107c89986ca4ec7adb56f731a567459, user_id=5bba915f4a374bba9c8f934b3f742b6c, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:01.296867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744fbf01-701d-4a5e-aced-cb3ea4fabfde admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:01.325889 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 936/1887] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:01 2022] PUT /identity/v3/projects/c107c89986ca4ec7adb56f731a567459/users/5bba915f4a374bba9c8f934b3f742b6c/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:01.330702 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3d60dd-90cb-494e-8f04-12cd994cdbea None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:01.331173 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3d60dd-90cb-494e-8f04-12cd994cdbea None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:01.331345 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3d60dd-90cb-494e-8f04-12cd994cdbea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:01.424461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-ca3d60dd-90cb-494e-8f04-12cd994cdbea None None] MFA Rules not processed for user `5bba915f4a374bba9c8f934b3f742b6c`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:01.425910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-ca3d60dd-90cb-494e-8f04-12cd994cdbea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:01.500228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:01.564680 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 952/1888] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:01 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:01.581121 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744fbf01-701d-4a5e-aced-cb3ea4fabfde admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:01.584887 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-744fbf01-701d-4a5e-aced-cb3ea4fabfde admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:01.650579 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3d60dd-90cb-494e-8f04-12cd994cdbea None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:01.653071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-ca3d60dd-90cb-494e-8f04-12cd994cdbea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:01.754031 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:01.831236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744fbf01-701d-4a5e-aced-cb3ea4fabfde admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:01.841625 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c8c851-f8ac-422d-a521-da2ce0d92204 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7fe9d21099e4c4891c28687bb673214', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:01.842210 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c8c851-f8ac-422d-a521-da2ce0d92204 service glance] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:01.842515 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c8c851-f8ac-422d-a521-da2ce0d92204 service glance] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:01.842797 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c8c851-f8ac-422d-a521-da2ce0d92204 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:01.870505 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c8c851-f8ac-422d-a521-da2ce0d92204 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:01.878049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c8c851-f8ac-422d-a521-da2ce0d92204 service glance] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:01.883797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3d60dd-90cb-494e-8f04-12cd994cdbea None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:01.896425 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53ed12f-9127-46a3-805e-454d39a229a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:01.897628 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53ed12f-9127-46a3-805e-454d39a229a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:01.898292 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53ed12f-9127-46a3-805e-454d39a229a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:01.900942 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53ed12f-9127-46a3-805e-454d39a229a6 admin admin] PATH_INFO: `/v3/projects/cffe27e20147449e873e664ea9e4816c` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:01.901104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 937/1889] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:01 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:01.903289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53ed12f-9127-46a3-805e-454d39a229a6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cffe27e20147449e873e664ea9e4816c)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:01.905291 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c8c851-f8ac-422d-a521-da2ce0d92204 service glance] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:01.909232 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-84c8c851-f8ac-422d-a521-da2ce0d92204 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:01.910302 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53ed12f-9127-46a3-805e-454d39a229a6 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:01.947606 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c8c851-f8ac-422d-a521-da2ce0d92204 service glance] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:01.961121 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fefe736-5709-4a06-920b-80dd0a088a80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:01.962052 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fefe736-5709-4a06-920b-80dd0a088a80 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:01.962197 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fefe736-5709-4a06-920b-80dd0a088a80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:01.962488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fefe736-5709-4a06-920b-80dd0a088a80 admin admin] PATH_INFO: `/v3/users/ed1c9b87ade9484fa37fe91bf1133b21` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:01.964578 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fefe736-5709-4a06-920b-80dd0a088a80 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ed1c9b87ade9484fa37fe91bf1133b21)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:01.973416 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fefe736-5709-4a06-920b-80dd0a088a80 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:01.994292 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d53ed12f-9127-46a3-805e-454d39a229a6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cffe27e20147449e873e664ea9e4816c', 'request_id': 'req-d53ed12f-9127-46a3-805e-454d39a229a6'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:02.013827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d53ed12f-9127-46a3-805e-454d39a229a6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cffe27e20147449e873e664ea9e4816c', 'request_id': 'req-d53ed12f-9127-46a3-805e-454d39a229a6'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:02.027185 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d53ed12f-9127-46a3-805e-454d39a229a6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cffe27e20147449e873e664ea9e4816c', 'request_id': 'req-d53ed12f-9127-46a3-805e-454d39a229a6'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:02.084615 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d53ed12f-9127-46a3-805e-454d39a229a6 admin admin] The token cache is being invalidate because project cffe27e20147449e873e664ea9e4816c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:02.087432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d53ed12f-9127-46a3-805e-454d39a229a6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:02.093149 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d53ed12f-9127-46a3-805e-454d39a229a6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:02.097048 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 953/1890] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:01 2022] DELETE /identity/v3/projects/cffe27e20147449e873e664ea9e4816c => generated 0 bytes in 447 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:02.103099 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4cb684-ee56-433e-82aa-ba70ebca3292 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:02.103513 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4cb684-ee56-433e-82aa-ba70ebca3292 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:02.103867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4cb684-ee56-433e-82aa-ba70ebca3292 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:02.137366 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9fefe736-5709-4a06-920b-80dd0a088a80 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ed1c9b87ade9484fa37fe91bf1133b21'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:02.166650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9fefe736-5709-4a06-920b-80dd0a088a80 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ed1c9b87ade9484fa37fe91bf1133b21'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:02.168080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9fefe736-5709-4a06-920b-80dd0a088a80 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ed1c9b87ade9484fa37fe91bf1133b21'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:02.169364 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9fefe736-5709-4a06-920b-80dd0a088a80 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ed1c9b87ade9484fa37fe91bf1133b21'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:02.181824 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9fefe736-5709-4a06-920b-80dd0a088a80 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ed1c9b87ade9484fa37fe91bf1133b21'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:02.185478 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-1a4cb684-ee56-433e-82aa-ba70ebca3292 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:02.193249 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-1a4cb684-ee56-433e-82aa-ba70ebca3292 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:02.216435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 938/1891] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:01 2022] DELETE /identity/v3/users/ed1c9b87ade9484fa37fe91bf1133b21 => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:02.230627 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45d0f42-1c44-4e88-af8f-1ce43e1e1205 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:02.234093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45d0f42-1c44-4e88-af8f-1ce43e1e1205 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:02.237798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45d0f42-1c44-4e88-af8f-1ce43e1e1205 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:02.297114 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-e45d0f42-1c44-4e88-af8f-1ce43e1e1205 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:02.298432 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-e45d0f42-1c44-4e88-af8f-1ce43e1e1205 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:02.448322 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 954/1892] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 348 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:58:02.451593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4cb684-ee56-433e-82aa-ba70ebca3292 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:02.463250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 939/1893] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:02.464249 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-1a4cb684-ee56-433e-82aa-ba70ebca3292 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:02.465426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496a6bc6-4363-4615-aea4-90f81cc63495 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:02.465881 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496a6bc6-4363-4615-aea4-90f81cc63495 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:02.466306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496a6bc6-4363-4615-aea4-90f81cc63495 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:02.496244 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:02.508416 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4cb684-ee56-433e-82aa-ba70ebca3292 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:02.531163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaae45de-b9c2-4da2-bc86-52a957d7ef5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:02.531821 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaae45de-b9c2-4da2-bc86-52a957d7ef5d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:02.532099 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaae45de-b9c2-4da2-bc86-52a957d7ef5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:02.532389 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaae45de-b9c2-4da2-bc86-52a957d7ef5d admin admin] PATH_INFO: `/v3/users/501378b46fb346fda42399e5d2af4ddc` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:02.535038 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaae45de-b9c2-4da2-bc86-52a957d7ef5d admin admin] RBAC: Authorizing `identity:delete_user(user_id=501378b46fb346fda42399e5d2af4ddc)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:02.539917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-496a6bc6-4363-4615-aea4-90f81cc63495 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:02.541487 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaae45de-b9c2-4da2-bc86-52a957d7ef5d admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:02.544005 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-496a6bc6-4363-4615-aea4-90f81cc63495 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:02.582543 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 940/1894] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:02.584130 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496a6bc6-4363-4615-aea4-90f81cc63495 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:02.586163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-496a6bc6-4363-4615-aea4-90f81cc63495 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:02.638035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-eaae45de-b9c2-4da2-bc86-52a957d7ef5d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '501378b46fb346fda42399e5d2af4ddc'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:02.653559 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-eaae45de-b9c2-4da2-bc86-52a957d7ef5d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '501378b46fb346fda42399e5d2af4ddc'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:02.661078 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:02.685166 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-eaae45de-b9c2-4da2-bc86-52a957d7ef5d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '501378b46fb346fda42399e5d2af4ddc'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:02.686500 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-eaae45de-b9c2-4da2-bc86-52a957d7ef5d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '501378b46fb346fda42399e5d2af4ddc'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:02.687987 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496a6bc6-4363-4615-aea4-90f81cc63495 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:02.693578 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d44b40-26a3-4ded-b5a1-5e5273b7e3c4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:02.693915 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d44b40-26a3-4ded-b5a1-5e5273b7e3c4 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:02.694485 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d44b40-26a3-4ded-b5a1-5e5273b7e3c4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:02.694776 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d44b40-26a3-4ded-b5a1-5e5273b7e3c4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:02.704700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-eaae45de-b9c2-4da2-bc86-52a957d7ef5d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '501378b46fb346fda42399e5d2af4ddc'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:02.711609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d44b40-26a3-4ded-b5a1-5e5273b7e3c4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:02.713359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d44b40-26a3-4ded-b5a1-5e5273b7e3c4 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:02.716281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 955/1895] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:02 2022] DELETE /identity/v3/users/501378b46fb346fda42399e5d2af4ddc => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:02.717675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaae45de-b9c2-4da2-bc86-52a957d7ef5d admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:02.721034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-eaae45de-b9c2-4da2-bc86-52a957d7ef5d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:02.732987 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 941/1896] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:02.734425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e17ded4-4b89-407b-bf87-23176af83728 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:02.734951 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e17ded4-4b89-407b-bf87-23176af83728 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:02.735212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e17ded4-4b89-407b-bf87-23176af83728 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:02.780253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-8e17ded4-4b89-407b-bf87-23176af83728 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:02.781727 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8e17ded4-4b89-407b-bf87-23176af83728 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:02.900356 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaae45de-b9c2-4da2-bc86-52a957d7ef5d admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:02.906369 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae61e13-dc8f-4a5d-8dab-4ebbe23f01cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:02.907325 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae61e13-dc8f-4a5d-8dab-4ebbe23f01cf admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:02.907762 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae61e13-dc8f-4a5d-8dab-4ebbe23f01cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:02.908899 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae61e13-dc8f-4a5d-8dab-4ebbe23f01cf admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:02.911123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae61e13-dc8f-4a5d-8dab-4ebbe23f01cf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:58:02.911543 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae61e13-dc8f-4a5d-8dab-4ebbe23f01cf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:02.915072 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae61e13-dc8f-4a5d-8dab-4ebbe23f01cf admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:02.935831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 956/1897] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:58:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:58:02.939557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae61e13-dc8f-4a5d-8dab-4ebbe23f01cf admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:02.944730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 942/1898] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:02.960672 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e17ded4-4b89-407b-bf87-23176af83728 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:02.962188 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8e17ded4-4b89-407b-bf87-23176af83728 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:02.963305 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae61e13-dc8f-4a5d-8dab-4ebbe23f01cf admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:02.971147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4de59e7-f373-4c0d-ac63-11742aeef8db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:02.971624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4de59e7-f373-4c0d-ac63-11742aeef8db admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:02.971933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4de59e7-f373-4c0d-ac63-11742aeef8db admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:02.972197 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4de59e7-f373-4c0d-ac63-11742aeef8db admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:02.972843 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4de59e7-f373-4c0d-ac63-11742aeef8db admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:02.975041 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4de59e7-f373-4c0d-ac63-11742aeef8db admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:03.022610 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 957/1899] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:02 2022] POST /identity/v3/projects => generated 387 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:03.026618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4de59e7-f373-4c0d-ac63-11742aeef8db admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:03.059700 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e17ded4-4b89-407b-bf87-23176af83728 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:03.085264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:03.109290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4de59e7-f373-4c0d-ac63-11742aeef8db admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:03.116669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e4ac7a-2c94-4d15-9a3e-63dffc1acfbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:03.117297 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e4ac7a-2c94-4d15-9a3e-63dffc1acfbe admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:03.117579 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e4ac7a-2c94-4d15-9a3e-63dffc1acfbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:03.117990 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e4ac7a-2c94-4d15-9a3e-63dffc1acfbe admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:03.118805 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e4ac7a-2c94-4d15-9a3e-63dffc1acfbe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:03.120894 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e4ac7a-2c94-4d15-9a3e-63dffc1acfbe admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:03.131143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ab0444-f471-47c0-92bd-e7bb3386db13 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:03.132017 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ab0444-f471-47c0-92bd-e7bb3386db13 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:03.132386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ab0444-f471-47c0-92bd-e7bb3386db13 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:03.132671 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ab0444-f471-47c0-92bd-e7bb3386db13 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:03.147616 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ab0444-f471-47c0-92bd-e7bb3386db13 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:03.149475 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ab0444-f471-47c0-92bd-e7bb3386db13 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:03.150173 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 958/1900] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:03 2022] POST /identity/v3/users => generated 294 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:03.151544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e4ac7a-2c94-4d15-9a3e-63dffc1acfbe admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:03.154318 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a0e4ac7a-2c94-4d15-9a3e-63dffc1acfbe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:03.162171 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 943/1901] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:03.163277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ab0444-f471-47c0-92bd-e7bb3386db13 service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:03.176717 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ab0444-f471-47c0-92bd-e7bb3386db13 service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:03.185143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df61915d-6180-4cc8-9dfc-caaa46221520 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:03.186048 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df61915d-6180-4cc8-9dfc-caaa46221520 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:03.186357 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df61915d-6180-4cc8-9dfc-caaa46221520 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:03.186726 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df61915d-6180-4cc8-9dfc-caaa46221520 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:03.187455 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df61915d-6180-4cc8-9dfc-caaa46221520 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:03.187775 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e4ac7a-2c94-4d15-9a3e-63dffc1acfbe admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:03.189196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df61915d-6180-4cc8-9dfc-caaa46221520 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:03.199905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fdb604-d890-4798-ac71-fc3a51fa7575 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:03.201267 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fdb604-d890-4798-ac71-fc3a51fa7575 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:03.201796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fdb604-d890-4798-ac71-fc3a51fa7575 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:03.202586 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fdb604-d890-4798-ac71-fc3a51fa7575 admin admin] PATH_INFO: `/v3/projects/454d03699de9403c9c3f7b05e7b5609e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:03.204458 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 944/1902] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:03 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:03.205079 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92fdb604-d890-4798-ac71-fc3a51fa7575 admin admin] RBAC: Authorizing `identity:delete_project(project_id=454d03699de9403c9c3f7b05e7b5609e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:03.205558 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df61915d-6180-4cc8-9dfc-caaa46221520 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:03.207551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92fdb604-d890-4798-ac71-fc3a51fa7575 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:03.222775 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df61915d-6180-4cc8-9dfc-caaa46221520 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:03.230090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe8f557-df0c-47f1-91cc-7d7187bf7b41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:03.230801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe8f557-df0c-47f1-91cc-7d7187bf7b41 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:03.231083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe8f557-df0c-47f1-91cc-7d7187bf7b41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:03.232023 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe8f557-df0c-47f1-91cc-7d7187bf7b41 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:03.232696 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe8f557-df0c-47f1-91cc-7d7187bf7b41 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:03.234328 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe8f557-df0c-47f1-91cc-7d7187bf7b41 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:03.250837 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 945/1903] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:03 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:03.257221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe8f557-df0c-47f1-91cc-7d7187bf7b41 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:03.277612 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe8f557-df0c-47f1-91cc-7d7187bf7b41 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:03.286789 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f8f721-4331-40b4-81b5-900872a4a871 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:03.287932 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f8f721-4331-40b4-81b5-900872a4a871 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:03.288256 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f8f721-4331-40b4-81b5-900872a4a871 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:03.288706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f8f721-4331-40b4-81b5-900872a4a871 admin admin] PATH_INFO: `/v3/projects/b39884ede1754058b67808b07532c0d1/users/32cd4a6e547f49eb93b7fda3444b541b/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:03.309281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-92fdb604-d890-4798-ac71-fc3a51fa7575 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '454d03699de9403c9c3f7b05e7b5609e', 'request_id': 'req-92fdb604-d890-4798-ac71-fc3a51fa7575'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:03.325366 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-92fdb604-d890-4798-ac71-fc3a51fa7575 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '454d03699de9403c9c3f7b05e7b5609e', 'request_id': 'req-92fdb604-d890-4798-ac71-fc3a51fa7575'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:03.332024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f8f721-4331-40b4-81b5-900872a4a871 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b39884ede1754058b67808b07532c0d1, user_id=32cd4a6e547f49eb93b7fda3444b541b, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:03.334813 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f8f721-4331-40b4-81b5-900872a4a871 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:03.339888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:03.345157 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-92fdb604-d890-4798-ac71-fc3a51fa7575 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '454d03699de9403c9c3f7b05e7b5609e', 'request_id': 'req-92fdb604-d890-4798-ac71-fc3a51fa7575'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:03.374769 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 946/1904] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:03 2022] PUT /identity/v3/projects/b39884ede1754058b67808b07532c0d1/users/32cd4a6e547f49eb93b7fda3444b541b/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:03.379693 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae57eaa4-aaf4-4fb0-a346-2d1683565440 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:03.381717 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae57eaa4-aaf4-4fb0-a346-2d1683565440 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:03.383669 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae57eaa4-aaf4-4fb0-a346-2d1683565440 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:03.385208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-92fdb604-d890-4798-ac71-fc3a51fa7575 admin admin] The token cache is being invalidate because project 454d03699de9403c9c3f7b05e7b5609e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:03.385659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-92fdb604-d890-4798-ac71-fc3a51fa7575 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:03.386883 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-92fdb604-d890-4798-ac71-fc3a51fa7575 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:03.388840 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 959/1905] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:03 2022] DELETE /identity/v3/projects/454d03699de9403c9c3f7b05e7b5609e => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:03.396410 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929a8c64-dea9-4a7a-96d0-a46cb54834da None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:03.396692 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929a8c64-dea9-4a7a-96d0-a46cb54834da None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:03.396967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929a8c64-dea9-4a7a-96d0-a46cb54834da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:03.435297 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-ae57eaa4-aaf4-4fb0-a346-2d1683565440 None None] MFA Rules not processed for user `32cd4a6e547f49eb93b7fda3444b541b`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:03.437308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ae57eaa4-aaf4-4fb0-a346-2d1683565440 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:03.451894 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-929a8c64-dea9-4a7a-96d0-a46cb54834da None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:03.453676 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-929a8c64-dea9-4a7a-96d0-a46cb54834da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:03.647260 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 947/1906] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:03 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 269 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:03.651914 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae57eaa4-aaf4-4fb0-a346-2d1683565440 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:03.655154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ae57eaa4-aaf4-4fb0-a346-2d1683565440 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:03.674223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 960/1907] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 279 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:03.703791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fb4536-e283-427d-90ed-89b2e743d92d None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:03.704140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fb4536-e283-427d-90ed-89b2e743d92d None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:03.704411 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fb4536-e283-427d-90ed-89b2e743d92d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:03.767367 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae57eaa4-aaf4-4fb0-a346-2d1683565440 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:03.781545 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-14fb4536-e283-427d-90ed-89b2e743d92d None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:03.783551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-14fb4536-e283-427d-90ed-89b2e743d92d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:03.786614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bb64e0-c8b5-4c3d-809e-0277fc6a8e4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:03.787246 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bb64e0-c8b5-4c3d-809e-0277fc6a8e4f admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:03.787521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bb64e0-c8b5-4c3d-809e-0277fc6a8e4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:03.787790 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bb64e0-c8b5-4c3d-809e-0277fc6a8e4f admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:03.788430 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79bb64e0-c8b5-4c3d-809e-0277fc6a8e4f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:03.790335 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79bb64e0-c8b5-4c3d-809e-0277fc6a8e4f admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:03.809700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 961/1908] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:03.811019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fb4536-e283-427d-90ed-89b2e743d92d None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:03.813906 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-14fb4536-e283-427d-90ed-89b2e743d92d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:03.835774 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fb4536-e283-427d-90ed-89b2e743d92d None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:03.841508 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf8716c-0ee5-4250-9585-75cf620c79b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:03.842371 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf8716c-0ee5-4250-9585-75cf620c79b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:03.843165 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf8716c-0ee5-4250-9585-75cf620c79b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:03.843448 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf8716c-0ee5-4250-9585-75cf620c79b8 admin admin] PATH_INFO: `/v3/projects/5e3c031bf2e041d5b3257c389ad48cdd` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:03.844614 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf8716c-0ee5-4250-9585-75cf620c79b8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5e3c031bf2e041d5b3257c389ad48cdd)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:03.846574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf8716c-0ee5-4250-9585-75cf620c79b8 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:03.852813 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 948/1909] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:03 2022] POST /identity/v3/projects => generated 387 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:03.854108 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bb64e0-c8b5-4c3d-809e-0277fc6a8e4f admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:03.936138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-abf8716c-0ee5-4250-9585-75cf620c79b8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5e3c031bf2e041d5b3257c389ad48cdd', 'request_id': 'req-abf8716c-0ee5-4250-9585-75cf620c79b8'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:03.941123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=85,keystone:INSERT=4,keystone:DELETE=11 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:03.956968 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-abf8716c-0ee5-4250-9585-75cf620c79b8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5e3c031bf2e041d5b3257c389ad48cdd', 'request_id': 'req-abf8716c-0ee5-4250-9585-75cf620c79b8'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:03.960826 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bb64e0-c8b5-4c3d-809e-0277fc6a8e4f admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:03.970538 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-abf8716c-0ee5-4250-9585-75cf620c79b8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5e3c031bf2e041d5b3257c389ad48cdd', 'request_id': 'req-abf8716c-0ee5-4250-9585-75cf620c79b8'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:03.990712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-abf8716c-0ee5-4250-9585-75cf620c79b8 admin admin] The token cache is being invalidate because project 5e3c031bf2e041d5b3257c389ad48cdd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:03.991084 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-abf8716c-0ee5-4250-9585-75cf620c79b8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:03.991708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-abf8716c-0ee5-4250-9585-75cf620c79b8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:03.993417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 962/1910] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:03 2022] DELETE /identity/v3/projects/5e3c031bf2e041d5b3257c389ad48cdd => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:03.994265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf8716c-0ee5-4250-9585-75cf620c79b8 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:03.995913 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-abf8716c-0ee5-4250-9585-75cf620c79b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:04.001145 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:04.095817 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b592f0-7080-4a40-88c6-eee6049cca49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:04.096449 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b592f0-7080-4a40-88c6-eee6049cca49 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:04.096835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b592f0-7080-4a40-88c6-eee6049cca49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:04.097183 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b592f0-7080-4a40-88c6-eee6049cca49 admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:04.097991 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b592f0-7080-4a40-88c6-eee6049cca49 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:58:04.098362 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b592f0-7080-4a40-88c6-eee6049cca49 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:04.101051 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b592f0-7080-4a40-88c6-eee6049cca49 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:04.105769 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf8716c-0ee5-4250-9585-75cf620c79b8 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:04.118272 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cecc615-1242-4f56-8482-7140905a4104 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:04.119040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cecc615-1242-4f56-8482-7140905a4104 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:04.119400 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cecc615-1242-4f56-8482-7140905a4104 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:04.119735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cecc615-1242-4f56-8482-7140905a4104 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:04.120489 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cecc615-1242-4f56-8482-7140905a4104 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:04.122933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cecc615-1242-4f56-8482-7140905a4104 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:04.133626 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 949/1911] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:58:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:58:04.135618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b592f0-7080-4a40-88c6-eee6049cca49 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:04.143845 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-54b592f0-7080-4a40-88c6-eee6049cca49 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:04.152110 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 963/1912] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:03 2022] POST /identity/v3/users => generated 300 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:04.153008 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cecc615-1242-4f56-8482-7140905a4104 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:04.155195 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-1cecc615-1242-4f56-8482-7140905a4104 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:04.160630 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b592f0-7080-4a40-88c6-eee6049cca49 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:04.167586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb784970-89e0-478e-b9d4-e57e084d5e40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:04.168229 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb784970-89e0-478e-b9d4-e57e084d5e40 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:04.168508 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb784970-89e0-478e-b9d4-e57e084d5e40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:04.168781 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb784970-89e0-478e-b9d4-e57e084d5e40 admin admin] PATH_INFO: `/v3/users/93bebdaa15b842d18e1ca569c46735e8` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:04.169899 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cecc615-1242-4f56-8482-7140905a4104 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:04.170018 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb784970-89e0-478e-b9d4-e57e084d5e40 admin admin] RBAC: Authorizing `identity:delete_user(user_id=93bebdaa15b842d18e1ca569c46735e8)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:04.171974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb784970-89e0-478e-b9d4-e57e084d5e40 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:04.175159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa712dcb-1b90-40f1-8165-b0149a6d6652 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:04.178491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa712dcb-1b90-40f1-8165-b0149a6d6652 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:04.178491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa712dcb-1b90-40f1-8165-b0149a6d6652 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:04.178491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa712dcb-1b90-40f1-8165-b0149a6d6652 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:04.178491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa712dcb-1b90-40f1-8165-b0149a6d6652 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:04.179715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa712dcb-1b90-40f1-8165-b0149a6d6652 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:04.213011 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 964/1913] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:04 2022] POST /identity/v3/projects => generated 399 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:04.215405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa712dcb-1b90-40f1-8165-b0149a6d6652 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:04.238217 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-eb784970-89e0-478e-b9d4-e57e084d5e40 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '93bebdaa15b842d18e1ca569c46735e8'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:04.248519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-eb784970-89e0-478e-b9d4-e57e084d5e40 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '93bebdaa15b842d18e1ca569c46735e8'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:04.249421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-eb784970-89e0-478e-b9d4-e57e084d5e40 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '93bebdaa15b842d18e1ca569c46735e8'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:04.250041 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-eb784970-89e0-478e-b9d4-e57e084d5e40 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '93bebdaa15b842d18e1ca569c46735e8'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:04.260208 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-eb784970-89e0-478e-b9d4-e57e084d5e40 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '93bebdaa15b842d18e1ca569c46735e8'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:04.285541 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 950/1914] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:04 2022] DELETE /identity/v3/users/93bebdaa15b842d18e1ca569c46735e8 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:04.287558 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb784970-89e0-478e-b9d4-e57e084d5e40 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:04.291723 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-eb784970-89e0-478e-b9d4-e57e084d5e40 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:04.306559 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa712dcb-1b90-40f1-8165-b0149a6d6652 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:04.415630 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb784970-89e0-478e-b9d4-e57e084d5e40 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:04.422605 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc38c929-f793-43b8-9905-efccaf39e9e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:04.423410 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc38c929-f793-43b8-9905-efccaf39e9e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:04.423803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc38c929-f793-43b8-9905-efccaf39e9e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:04.424644 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc38c929-f793-43b8-9905-efccaf39e9e6 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:04.424848 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc38c929-f793-43b8-9905-efccaf39e9e6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:04.427372 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:DELETE=3,keystone:SELECT=92,keystone:INSERT=5 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:04.429532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc38c929-f793-43b8-9905-efccaf39e9e6 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:04.448373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28bbb95c-d8ba-4ab9-8035-8d5fe8ec0a5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:04.449666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bbb95c-d8ba-4ab9-8035-8d5fe8ec0a5c admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:04.450139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bbb95c-d8ba-4ab9-8035-8d5fe8ec0a5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:04.450593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bbb95c-d8ba-4ab9-8035-8d5fe8ec0a5c admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:04.451618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28bbb95c-d8ba-4ab9-8035-8d5fe8ec0a5c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:04.454010 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28bbb95c-d8ba-4ab9-8035-8d5fe8ec0a5c admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:04.475102 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 965/1915] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:04 2022] GET /identity/v3/roles => generated 1420 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:04.477083 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28bbb95c-d8ba-4ab9-8035-8d5fe8ec0a5c admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:04.479619 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 951/1916] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:04 2022] POST /identity/v3/users => generated 300 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:04.483767 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc38c929-f793-43b8-9905-efccaf39e9e6 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:04.484833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-28bbb95c-d8ba-4ab9-8035-8d5fe8ec0a5c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:04.505308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28bbb95c-d8ba-4ab9-8035-8d5fe8ec0a5c admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:04.515223 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc38c929-f793-43b8-9905-efccaf39e9e6 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:04.528818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235b94d0-747c-4b6c-acf1-16ef4bc69935 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:04.529573 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235b94d0-747c-4b6c-acf1-16ef4bc69935 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:04.529855 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235b94d0-747c-4b6c-acf1-16ef4bc69935 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:04.530127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235b94d0-747c-4b6c-acf1-16ef4bc69935 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:04.530906 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235b94d0-747c-4b6c-acf1-16ef4bc69935 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:04.533233 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235b94d0-747c-4b6c-acf1-16ef4bc69935 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:04.535202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882ca5fa-3e36-4041-9bd0-a9fc6102328e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:04.538044 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882ca5fa-3e36-4041-9bd0-a9fc6102328e admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:04.541131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882ca5fa-3e36-4041-9bd0-a9fc6102328e admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:04.541531 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882ca5fa-3e36-4041-9bd0-a9fc6102328e admin admin] PATH_INFO: `/v3/projects/ce5bba490b034631900a08a305fb41a9/users/a0db1db80ec04a08a3e5d047e1b40c97/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:04.578842 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 952/1917] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:04 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:04.580111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235b94d0-747c-4b6c-acf1-16ef4bc69935 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:04.582022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-882ca5fa-3e36-4041-9bd0-a9fc6102328e admin admin] RBAC: Authorizing `identity:create_grant(project_id=ce5bba490b034631900a08a305fb41a9, user_id=a0db1db80ec04a08a3e5d047e1b40c97, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:04.586276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:04.588432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-882ca5fa-3e36-4041-9bd0-a9fc6102328e admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:04.601219 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235b94d0-747c-4b6c-acf1-16ef4bc69935 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:04.616471 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ed508b-22d3-42c0-8919-4e319c803c54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:04.619049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ed508b-22d3-42c0-8919-4e319c803c54 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:04.619471 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ed508b-22d3-42c0-8919-4e319c803c54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:04.620515 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ed508b-22d3-42c0-8919-4e319c803c54 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:04.621359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ed508b-22d3-42c0-8919-4e319c803c54 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:04.623197 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ed508b-22d3-42c0-8919-4e319c803c54 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:04.627233 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 966/1918] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:04 2022] PUT /identity/v3/projects/ce5bba490b034631900a08a305fb41a9/users/a0db1db80ec04a08a3e5d047e1b40c97/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:04.630595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882ca5fa-3e36-4041-9bd0-a9fc6102328e admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:04.652216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 953/1919] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:04 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:04.666691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ed508b-22d3-42c0-8919-4e319c803c54 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:04.695936 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ed508b-22d3-42c0-8919-4e319c803c54 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:04.742431 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882ca5fa-3e36-4041-9bd0-a9fc6102328e admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:04.748556 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7748e987-dfbd-478f-8666-209a64028cba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:04.749806 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7748e987-dfbd-478f-8666-209a64028cba admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:04.750251 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7748e987-dfbd-478f-8666-209a64028cba admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:04.750893 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7748e987-dfbd-478f-8666-209a64028cba admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:04.751600 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7748e987-dfbd-478f-8666-209a64028cba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:04.753333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7748e987-dfbd-478f-8666-209a64028cba admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:04.764699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 967/1920] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:04 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:58:04.768855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7748e987-dfbd-478f-8666-209a64028cba admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:04.783004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7748e987-dfbd-478f-8666-209a64028cba admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:04.789767 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f5a6ea-eebb-4602-b4ee-9df801151efa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:04.794359 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f5a6ea-eebb-4602-b4ee-9df801151efa admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:04.794359 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f5a6ea-eebb-4602-b4ee-9df801151efa admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:04.794359 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f5a6ea-eebb-4602-b4ee-9df801151efa admin admin] PATH_INFO: `/v3/projects/ce5bba490b034631900a08a305fb41a9/users/a0db1db80ec04a08a3e5d047e1b40c97/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:04.794359 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f5a6ea-eebb-4602-b4ee-9df801151efa admin admin] RBAC: Authorizing `identity:create_grant(project_id=ce5bba490b034631900a08a305fb41a9, user_id=a0db1db80ec04a08a3e5d047e1b40c97, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:04.797817 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f5a6ea-eebb-4602-b4ee-9df801151efa admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:04.807847 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490c1c2a-6339-44c5-9049-8a39988c6c2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:04.808689 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490c1c2a-6339-44c5-9049-8a39988c6c2c admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:04.808881 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490c1c2a-6339-44c5-9049-8a39988c6c2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:04.809152 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490c1c2a-6339-44c5-9049-8a39988c6c2c admin admin] PATH_INFO: `/v3/projects/fed6c500a3bf409a8de067697b583a59/users/6d507dc6ddd04533a4c570714a5fa996/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:04.823362 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 968/1921] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:04 2022] PUT /identity/v3/projects/ce5bba490b034631900a08a305fb41a9/users/a0db1db80ec04a08a3e5d047e1b40c97/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:04.824715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f5a6ea-eebb-4602-b4ee-9df801151efa admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:04.826259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-47f5a6ea-eebb-4602-b4ee-9df801151efa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:04.839001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490c1c2a-6339-44c5-9049-8a39988c6c2c admin admin] RBAC: Authorizing `identity:create_grant(project_id=fed6c500a3bf409a8de067697b583a59, user_id=6d507dc6ddd04533a4c570714a5fa996, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:04.850881 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490c1c2a-6339-44c5-9049-8a39988c6c2c admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:04.865152 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 954/1922] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:04 2022] PUT /identity/v3/projects/fed6c500a3bf409a8de067697b583a59/users/6d507dc6ddd04533a4c570714a5fa996/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:04.867145 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b523aa92-38a6-4610-9523-f7ff6300de57 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:04.867507 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b523aa92-38a6-4610-9523-f7ff6300de57 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:04.867849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b523aa92-38a6-4610-9523-f7ff6300de57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:04.906551 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-b523aa92-38a6-4610-9523-f7ff6300de57 None None] MFA Rules not processed for user `a0db1db80ec04a08a3e5d047e1b40c97`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:04.908095 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-b523aa92-38a6-4610-9523-f7ff6300de57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:04.924257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f5a6ea-eebb-4602-b4ee-9df801151efa admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:05.009400 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:05.030975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6eac2d-2259-4c9b-9c7c-7d2bb0acc8a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:05.030975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6eac2d-2259-4c9b-9c7c-7d2bb0acc8a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:05.030975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6eac2d-2259-4c9b-9c7c-7d2bb0acc8a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:05.030975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6eac2d-2259-4c9b-9c7c-7d2bb0acc8a6 admin admin] PATH_INFO: `/v3/projects/4bb885f46fb24f71b8b03c593ee4225b` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:05.031836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 955/1923] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:04 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:05.031836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f96664-e271-4a0b-abfb-fa93e46d6295 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:05.031836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f96664-e271-4a0b-abfb-fa93e46d6295 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:05.032371 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6eac2d-2259-4c9b-9c7c-7d2bb0acc8a6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4bb885f46fb24f71b8b03c593ee4225b)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:05.032517 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f96664-e271-4a0b-abfb-fa93e46d6295 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:05.032655 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6eac2d-2259-4c9b-9c7c-7d2bb0acc8a6 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:05.068519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-78f96664-e271-4a0b-abfb-fa93e46d6295 None None] MFA Rules not processed for user `6d507dc6ddd04533a4c570714a5fa996`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:05.069855 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-78f96664-e271-4a0b-abfb-fa93e46d6295 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:05.113996 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-6c6eac2d-2259-4c9b-9c7c-7d2bb0acc8a6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4bb885f46fb24f71b8b03c593ee4225b', 'request_id': 'req-6c6eac2d-2259-4c9b-9c7c-7d2bb0acc8a6'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:05.124114 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-6c6eac2d-2259-4c9b-9c7c-7d2bb0acc8a6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4bb885f46fb24f71b8b03c593ee4225b', 'request_id': 'req-6c6eac2d-2259-4c9b-9c7c-7d2bb0acc8a6'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:05.142310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:05.146494 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-6c6eac2d-2259-4c9b-9c7c-7d2bb0acc8a6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4bb885f46fb24f71b8b03c593ee4225b', 'request_id': 'req-6c6eac2d-2259-4c9b-9c7c-7d2bb0acc8a6'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:05.169753 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-6c6eac2d-2259-4c9b-9c7c-7d2bb0acc8a6 admin admin] The token cache is being invalidate because project 4bb885f46fb24f71b8b03c593ee4225b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:05.170320 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 956/1924] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:05 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:05.172971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f96664-e271-4a0b-abfb-fa93e46d6295 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:05.174073 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-6c6eac2d-2259-4c9b-9c7c-7d2bb0acc8a6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:05.175812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-6c6eac2d-2259-4c9b-9c7c-7d2bb0acc8a6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:05.183029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-78f96664-e271-4a0b-abfb-fa93e46d6295 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:05.191510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 969/1925] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:04 2022] DELETE /identity/v3/projects/4bb885f46fb24f71b8b03c593ee4225b => generated 0 bytes in 363 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:05.193754 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6eac2d-2259-4c9b-9c7c-7d2bb0acc8a6 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:05.195521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-6c6eac2d-2259-4c9b-9c7c-7d2bb0acc8a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:05.311147 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f96664-e271-4a0b-abfb-fa93e46d6295 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:05.318137 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6eac2d-2259-4c9b-9c7c-7d2bb0acc8a6 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:05.318688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f67c526-2a21-4c2a-a2c7-d3fc6ae3c7f9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:05.323002 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f67c526-2a21-4c2a-a2c7-d3fc6ae3c7f9 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:05.323002 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f67c526-2a21-4c2a-a2c7-d3fc6ae3c7f9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:05.323002 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f67c526-2a21-4c2a-a2c7-d3fc6ae3c7f9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:05.325036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-1f67c526-2a21-4c2a-a2c7-d3fc6ae3c7f9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:05.326492 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5c69ab-4ab2-44e3-95ff-69b6b4ad5ec2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:05.327155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5c69ab-4ab2-44e3-95ff-69b6b4ad5ec2 service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:05.327431 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5c69ab-4ab2-44e3-95ff-69b6b4ad5ec2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:05.327779 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5c69ab-4ab2-44e3-95ff-69b6b4ad5ec2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:05.330145 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-2b5c69ab-4ab2-44e3-95ff-69b6b4ad5ec2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:05.476603 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f67c526-2a21-4c2a-a2c7-d3fc6ae3c7f9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:05.478615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f67c526-2a21-4c2a-a2c7-d3fc6ae3c7f9 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:05.491967 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 957/1926] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:05 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:05.492729 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4ae1ab-2505-40d7-8130-afba00531a3c None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:05.493163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4ae1ab-2505-40d7-8130-afba00531a3c None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:05.493576 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4ae1ab-2505-40d7-8130-afba00531a3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:05.502113 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5c69ab-4ab2-44e3-95ff-69b6b4ad5ec2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:05.504073 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5c69ab-4ab2-44e3-95ff-69b6b4ad5ec2 service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:05.517840 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 970/1927] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:05 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:05.533123 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-1e4ae1ab-2505-40d7-8130-afba00531a3c None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:05.538629 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-1e4ae1ab-2505-40d7-8130-afba00531a3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:05.625815 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 958/1928] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:05.647153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3ae133-fde8-45ea-a1bb-dd320c7ba197 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:05.649167 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3ae133-fde8-45ea-a1bb-dd320c7ba197 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:05.649167 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3ae133-fde8-45ea-a1bb-dd320c7ba197 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:05.702028 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-ef3ae133-fde8-45ea-a1bb-dd320c7ba197 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:05.703281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-ef3ae133-fde8-45ea-a1bb-dd320c7ba197 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:05.732701 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 971/1929] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:05.736041 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4ae1ab-2505-40d7-8130-afba00531a3c None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:05.751598 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4ae1ab-2505-40d7-8130-afba00531a3c None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:05.761142 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5786f7a7-8c57-4168-9fdc-4907e825463f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:05.762041 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5786f7a7-8c57-4168-9fdc-4907e825463f admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:05.762373 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5786f7a7-8c57-4168-9fdc-4907e825463f admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:05.762852 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5786f7a7-8c57-4168-9fdc-4907e825463f admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:05.763624 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5786f7a7-8c57-4168-9fdc-4907e825463f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:58:05.764071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5786f7a7-8c57-4168-9fdc-4907e825463f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:05.767909 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5786f7a7-8c57-4168-9fdc-4907e825463f admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:05.769621 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3ae133-fde8-45ea-a1bb-dd320c7ba197 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:05.771176 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-ef3ae133-fde8-45ea-a1bb-dd320c7ba197 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:05.792213 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3ae133-fde8-45ea-a1bb-dd320c7ba197 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:05.796016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 959/1930] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:58:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:58:05.801579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5786f7a7-8c57-4168-9fdc-4907e825463f admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:05.811256 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e40e08b-4669-47da-b476-485737469093 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:05.811673 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e40e08b-4669-47da-b476-485737469093 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:05.811967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e40e08b-4669-47da-b476-485737469093 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:05.812233 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e40e08b-4669-47da-b476-485737469093 admin admin] PATH_INFO: `/v3/users/32cd4a6e547f49eb93b7fda3444b541b` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:05.817676 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5786f7a7-8c57-4168-9fdc-4907e825463f admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:05.818272 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e40e08b-4669-47da-b476-485737469093 admin admin] RBAC: Authorizing `identity:delete_user(user_id=32cd4a6e547f49eb93b7fda3444b541b)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:05.821300 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e40e08b-4669-47da-b476-485737469093 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:05.828415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53abba0-f3bd-4010-aa97-47719dbb7dba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:05.829436 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53abba0-f3bd-4010-aa97-47719dbb7dba admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:05.830013 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53abba0-f3bd-4010-aa97-47719dbb7dba admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:05.830448 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53abba0-f3bd-4010-aa97-47719dbb7dba admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:05.831591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53abba0-f3bd-4010-aa97-47719dbb7dba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:05.834428 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53abba0-f3bd-4010-aa97-47719dbb7dba admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:05.893003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:05.912949 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 960/1931] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:05 2022] POST /identity/v3/projects => generated 363 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:05.917687 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53abba0-f3bd-4010-aa97-47719dbb7dba admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:05.933743 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:05.951271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2e40e08b-4669-47da-b476-485737469093 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '32cd4a6e547f49eb93b7fda3444b541b'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:05.966904 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2e40e08b-4669-47da-b476-485737469093 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '32cd4a6e547f49eb93b7fda3444b541b'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:05.992936 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2e40e08b-4669-47da-b476-485737469093 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '32cd4a6e547f49eb93b7fda3444b541b'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:05.994795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2e40e08b-4669-47da-b476-485737469093 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '32cd4a6e547f49eb93b7fda3444b541b'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:06.009978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2e40e08b-4669-47da-b476-485737469093 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '32cd4a6e547f49eb93b7fda3444b541b'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:06.043861 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 972/1932] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:05 2022] DELETE /identity/v3/users/32cd4a6e547f49eb93b7fda3444b541b => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:06.049195 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4565c1b7-0216-45c2-94f8-5d1fbb04108f None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:06.049483 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4565c1b7-0216-45c2-94f8-5d1fbb04108f None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:06.049911 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4565c1b7-0216-45c2-94f8-5d1fbb04108f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:06.088748 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-4565c1b7-0216-45c2-94f8-5d1fbb04108f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:06.091104 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-4565c1b7-0216-45c2-94f8-5d1fbb04108f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:06.106483 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53abba0-f3bd-4010-aa97-47719dbb7dba admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:06.187353 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324e1063-3355-48fc-85fe-69693a443df8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:06.188301 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324e1063-3355-48fc-85fe-69693a443df8 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:06.188783 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324e1063-3355-48fc-85fe-69693a443df8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:06.189070 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324e1063-3355-48fc-85fe-69693a443df8 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:06.189740 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-324e1063-3355-48fc-85fe-69693a443df8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:06.194504 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 973/1933] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:06.195352 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-324e1063-3355-48fc-85fe-69693a443df8 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:06.202083 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4565c1b7-0216-45c2-94f8-5d1fbb04108f None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:06.204529 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-4565c1b7-0216-45c2-94f8-5d1fbb04108f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:06.218506 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 961/1934] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:05 2022] POST /identity/v3/users => generated 282 bytes in 301 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:06.222544 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324e1063-3355-48fc-85fe-69693a443df8 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:06.226289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-324e1063-3355-48fc-85fe-69693a443df8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:06.241572 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324e1063-3355-48fc-85fe-69693a443df8 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:06.248151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e27629-fdc6-4003-a1f7-3b24a49052b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:06.248810 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e27629-fdc6-4003-a1f7-3b24a49052b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:06.249090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e27629-fdc6-4003-a1f7-3b24a49052b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:06.249392 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e27629-fdc6-4003-a1f7-3b24a49052b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:06.250049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e27629-fdc6-4003-a1f7-3b24a49052b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:06.251710 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e27629-fdc6-4003-a1f7-3b24a49052b3 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:06.264152 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 962/1935] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:06 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:58:06.270543 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e27629-fdc6-4003-a1f7-3b24a49052b3 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:06.283985 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e27629-fdc6-4003-a1f7-3b24a49052b3 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:06.292812 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c82abd5-49c2-4ccb-9074-efe3cc9dc159 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:06.294083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c82abd5-49c2-4ccb-9074-efe3cc9dc159 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:06.294867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c82abd5-49c2-4ccb-9074-efe3cc9dc159 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:06.295320 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c82abd5-49c2-4ccb-9074-efe3cc9dc159 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:06.296219 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c82abd5-49c2-4ccb-9074-efe3cc9dc159 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:06.298394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c82abd5-49c2-4ccb-9074-efe3cc9dc159 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:06.315033 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4565c1b7-0216-45c2-94f8-5d1fbb04108f None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:06.320629 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56261a1-eb85-43c8-9110-8c5a5946394e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:06.321940 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56261a1-eb85-43c8-9110-8c5a5946394e service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:06.322626 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56261a1-eb85-43c8-9110-8c5a5946394e service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:06.323262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56261a1-eb85-43c8-9110-8c5a5946394e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:06.339216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 963/1936] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:06 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:06.339593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56261a1-eb85-43c8-9110-8c5a5946394e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:06.340425 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56261a1-eb85-43c8-9110-8c5a5946394e service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:06.341356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c82abd5-49c2-4ccb-9074-efe3cc9dc159 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:06.352629 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 974/1937] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:58:06.363285 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c82abd5-49c2-4ccb-9074-efe3cc9dc159 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:06.371153 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8704ab-c746-45f7-b84d-3c27e3177d25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:06.372069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8704ab-c746-45f7-b84d-3c27e3177d25 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:06.372533 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8704ab-c746-45f7-b84d-3c27e3177d25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:06.373084 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8704ab-c746-45f7-b84d-3c27e3177d25 admin admin] PATH_INFO: `/v3/projects/d40411f11d0740b3946a1f8fdded43f6/users/50fb1afce26a47a9848da001454a23b0/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:06.411106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8704ab-c746-45f7-b84d-3c27e3177d25 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d40411f11d0740b3946a1f8fdded43f6, user_id=50fb1afce26a47a9848da001454a23b0, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:06.418491 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8704ab-c746-45f7-b84d-3c27e3177d25 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:06.458909 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 964/1938] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:06 2022] PUT /identity/v3/projects/d40411f11d0740b3946a1f8fdded43f6/users/50fb1afce26a47a9848da001454a23b0/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:06.465997 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ca259d-e28e-4de0-b1f5-5dd05310d9f9 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:06.466572 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ca259d-e28e-4de0-b1f5-5dd05310d9f9 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:06.466701 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ca259d-e28e-4de0-b1f5-5dd05310d9f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:06.530575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-a7ca259d-e28e-4de0-b1f5-5dd05310d9f9 None None] MFA Rules not processed for user `50fb1afce26a47a9848da001454a23b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:06.535522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a7ca259d-e28e-4de0-b1f5-5dd05310d9f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:06.601424 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:06.639082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 975/1939] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:06 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:06.643699 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8704ab-c746-45f7-b84d-3c27e3177d25 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:06.728139 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:06.737401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8704ab-c746-45f7-b84d-3c27e3177d25 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:06.745201 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d1fce6-8943-4dd1-8f4b-80fc3da410f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:06.745870 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d1fce6-8943-4dd1-8f4b-80fc3da410f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:06.746319 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d1fce6-8943-4dd1-8f4b-80fc3da410f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:06.746817 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d1fce6-8943-4dd1-8f4b-80fc3da410f1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:06.747690 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d1fce6-8943-4dd1-8f4b-80fc3da410f1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:06.749767 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d1fce6-8943-4dd1-8f4b-80fc3da410f1 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:06.789381 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 965/1940] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:06 2022] POST /identity/v3/projects => generated 363 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:06.790905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ca259d-e28e-4de0-b1f5-5dd05310d9f9 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:06.809025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ca259d-e28e-4de0-b1f5-5dd05310d9f9 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:06.876466 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d1fce6-8943-4dd1-8f4b-80fc3da410f1 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:06.878761 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c7d1fce6-8943-4dd1-8f4b-80fc3da410f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:06.913218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3cd9c30-1cda-466b-bd75-166c73079cc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:06.913977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3cd9c30-1cda-466b-bd75-166c73079cc5 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:06.914259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3cd9c30-1cda-466b-bd75-166c73079cc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:06.914567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3cd9c30-1cda-466b-bd75-166c73079cc5 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:06.915448 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3cd9c30-1cda-466b-bd75-166c73079cc5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:06.917367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3cd9c30-1cda-466b-bd75-166c73079cc5 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:06.941517 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 976/1941] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:06 2022] POST /identity/v3/users => generated 288 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:06.945144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3cd9c30-1cda-466b-bd75-166c73079cc5 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:06.955019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3cd9c30-1cda-466b-bd75-166c73079cc5 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:06.957145 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d1fce6-8943-4dd1-8f4b-80fc3da410f1 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:06.961145 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11dd66b-1884-460d-9b8b-6170b1220150 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:06.961748 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11dd66b-1884-460d-9b8b-6170b1220150 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:06.962027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11dd66b-1884-460d-9b8b-6170b1220150 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:06.962295 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11dd66b-1884-460d-9b8b-6170b1220150 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:06.962985 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11dd66b-1884-460d-9b8b-6170b1220150 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:06.964301 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddac86c-6572-4ef4-a34e-8336c2c48254 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:06.964562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11dd66b-1884-460d-9b8b-6170b1220150 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:06.964944 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddac86c-6572-4ef4-a34e-8336c2c48254 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:06.965251 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddac86c-6572-4ef4-a34e-8336c2c48254 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:06.965529 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddac86c-6572-4ef4-a34e-8336c2c48254 admin admin] PATH_INFO: `/v3/projects/b39884ede1754058b67808b07532c0d1` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:06.966755 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddac86c-6572-4ef4-a34e-8336c2c48254 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b39884ede1754058b67808b07532c0d1)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:06.968823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddac86c-6572-4ef4-a34e-8336c2c48254 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:06.975481 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 977/1942] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:06 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:58:06.979756 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11dd66b-1884-460d-9b8b-6170b1220150 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:06.990634 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11dd66b-1884-460d-9b8b-6170b1220150 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:06.995447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a810c0-af07-42e7-9c0e-3df9c04ae540 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:06.996341 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a810c0-af07-42e7-9c0e-3df9c04ae540 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:06.996961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a810c0-af07-42e7-9c0e-3df9c04ae540 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:06.996961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a810c0-af07-42e7-9c0e-3df9c04ae540 admin admin] PATH_INFO: `/v3/projects/6b3085ac38454731ae03e679739e1503/users/7db453a24c5c48fc9845f5a544e12b49/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:07.017651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a810c0-af07-42e7-9c0e-3df9c04ae540 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6b3085ac38454731ae03e679739e1503, user_id=7db453a24c5c48fc9845f5a544e12b49, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:07.021220 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a810c0-af07-42e7-9c0e-3df9c04ae540 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:07.026874 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-fddac86c-6572-4ef4-a34e-8336c2c48254 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b39884ede1754058b67808b07532c0d1', 'request_id': 'req-fddac86c-6572-4ef4-a34e-8336c2c48254'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:07.033114 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-fddac86c-6572-4ef4-a34e-8336c2c48254 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b39884ede1754058b67808b07532c0d1', 'request_id': 'req-fddac86c-6572-4ef4-a34e-8336c2c48254'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:07.049148 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 978/1943] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:06 2022] PUT /identity/v3/projects/6b3085ac38454731ae03e679739e1503/users/7db453a24c5c48fc9845f5a544e12b49/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:07.053697 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-fddac86c-6572-4ef4-a34e-8336c2c48254 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b39884ede1754058b67808b07532c0d1', 'request_id': 'req-fddac86c-6572-4ef4-a34e-8336c2c48254'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:07.054110 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a810c0-af07-42e7-9c0e-3df9c04ae540 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:07.082917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-fddac86c-6572-4ef4-a34e-8336c2c48254 admin admin] The token cache is being invalidate because project b39884ede1754058b67808b07532c0d1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:07.083269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-fddac86c-6572-4ef4-a34e-8336c2c48254 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:07.085263 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-fddac86c-6572-4ef4-a34e-8336c2c48254 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:07.088714 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 966/1944] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:06 2022] DELETE /identity/v3/projects/b39884ede1754058b67808b07532c0d1 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:07.093496 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddac86c-6572-4ef4-a34e-8336c2c48254 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:07.094100 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-fddac86c-6572-4ef4-a34e-8336c2c48254 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:07.157100 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a810c0-af07-42e7-9c0e-3df9c04ae540 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:07.165167 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8ccb81-862f-4cee-99c1-a05c15df8caf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:07.165975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8ccb81-862f-4cee-99c1-a05c15df8caf admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:07.166325 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8ccb81-862f-4cee-99c1-a05c15df8caf admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:07.166624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8ccb81-862f-4cee-99c1-a05c15df8caf admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:07.167601 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8ccb81-862f-4cee-99c1-a05c15df8caf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:07.168943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8ccb81-862f-4cee-99c1-a05c15df8caf admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:07.182895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 979/1945] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:07 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:07.184083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddac86c-6572-4ef4-a34e-8336c2c48254 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:07.185365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8ccb81-862f-4cee-99c1-a05c15df8caf admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:07.187870 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-5e8ccb81-862f-4cee-99c1-a05c15df8caf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:07.189415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b46c64b-f3b3-43ef-a7c1-6668d4149e32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:07.190069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b46c64b-f3b3-43ef-a7c1-6668d4149e32 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:07.190352 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b46c64b-f3b3-43ef-a7c1-6668d4149e32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:07.190664 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b46c64b-f3b3-43ef-a7c1-6668d4149e32 admin admin] PATH_INFO: `/v3/users/a0db1db80ec04a08a3e5d047e1b40c97` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:07.191919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b46c64b-f3b3-43ef-a7c1-6668d4149e32 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a0db1db80ec04a08a3e5d047e1b40c97)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:07.194499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b46c64b-f3b3-43ef-a7c1-6668d4149e32 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:07.203291 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8ccb81-862f-4cee-99c1-a05c15df8caf admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:07.208832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850102c8-0089-4d38-b205-d24355a749c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:07.209511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850102c8-0089-4d38-b205-d24355a749c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:07.210046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850102c8-0089-4d38-b205-d24355a749c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:07.210046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850102c8-0089-4d38-b205-d24355a749c1 admin admin] PATH_INFO: `/v3/projects/6b3085ac38454731ae03e679739e1503/users/7db453a24c5c48fc9845f5a544e12b49/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:07.211282 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:07.213029 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850102c8-0089-4d38-b205-d24355a749c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6b3085ac38454731ae03e679739e1503, user_id=7db453a24c5c48fc9845f5a544e12b49, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:07.221070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850102c8-0089-4d38-b205-d24355a749c1 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:07.236933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 980/1946] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:07 2022] PUT /identity/v3/projects/6b3085ac38454731ae03e679739e1503/users/7db453a24c5c48fc9845f5a544e12b49/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:07.247851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d1e8e6-eb40-41f0-ab2a-04464e064c1f None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:07.248247 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d1e8e6-eb40-41f0-ab2a-04464e064c1f None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:07.248643 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d1e8e6-eb40-41f0-ab2a-04464e064c1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:07.271375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6b46c64b-f3b3-43ef-a7c1-6668d4149e32 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a0db1db80ec04a08a3e5d047e1b40c97'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:07.281181 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6b46c64b-f3b3-43ef-a7c1-6668d4149e32 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a0db1db80ec04a08a3e5d047e1b40c97'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:07.281972 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6b46c64b-f3b3-43ef-a7c1-6668d4149e32 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a0db1db80ec04a08a3e5d047e1b40c97'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:07.282697 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6b46c64b-f3b3-43ef-a7c1-6668d4149e32 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a0db1db80ec04a08a3e5d047e1b40c97'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:07.285677 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-65d1e8e6-eb40-41f0-ab2a-04464e064c1f None None] MFA Rules not processed for user `7db453a24c5c48fc9845f5a544e12b49`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:07.287454 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-65d1e8e6-eb40-41f0-ab2a-04464e064c1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:07.291573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:07.291831 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6b46c64b-f3b3-43ef-a7c1-6668d4149e32 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a0db1db80ec04a08a3e5d047e1b40c97'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:07.312804 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 967/1947] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:07 2022] DELETE /identity/v3/users/a0db1db80ec04a08a3e5d047e1b40c97 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:07.392552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 981/1948] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:07 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:07.407465 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b46c64b-f3b3-43ef-a7c1-6668d4149e32 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:07.409846 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-6b46c64b-f3b3-43ef-a7c1-6668d4149e32 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:07.490170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b46c64b-f3b3-43ef-a7c1-6668d4149e32 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:07.499611 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde90e17-c53c-4be8-8296-04f3594e2c6c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:07.500353 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde90e17-c53c-4be8-8296-04f3594e2c6c service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:07.501417 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde90e17-c53c-4be8-8296-04f3594e2c6c service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:07.501747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde90e17-c53c-4be8-8296-04f3594e2c6c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:07.602494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde90e17-c53c-4be8-8296-04f3594e2c6c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:07.603575 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde90e17-c53c-4be8-8296-04f3594e2c6c service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:07.615833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 968/1949] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:07 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 209 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:07.654514 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d1e8e6-eb40-41f0-ab2a-04464e064c1f None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:07.655308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-65d1e8e6-eb40-41f0-ab2a-04464e064c1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:07.753729 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde90e17-c53c-4be8-8296-04f3594e2c6c service cinder] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:07.770446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde90e17-c53c-4be8-8296-04f3594e2c6c service cinder] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:07.775198 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff91a347-c488-48ae-a628-7d86cc082cda service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:07.776114 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff91a347-c488-48ae-a628-7d86cc082cda service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:07.776489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff91a347-c488-48ae-a628-7d86cc082cda service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:07.776830 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff91a347-c488-48ae-a628-7d86cc082cda service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:07.779622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ff91a347-c488-48ae-a628-7d86cc082cda service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:07.789156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d1e8e6-eb40-41f0-ab2a-04464e064c1f None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:07.859749 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:07.889378 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff91a347-c488-48ae-a628-7d86cc082cda service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:07.890963 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff91a347-c488-48ae-a628-7d86cc082cda service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:07.903266 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 969/1950] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:07 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:07.958802 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35497de4-abad-429e-b583-0d17c0937b1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:07.962316 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35497de4-abad-429e-b583-0d17c0937b1f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:07.963483 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35497de4-abad-429e-b583-0d17c0937b1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:07.963877 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35497de4-abad-429e-b583-0d17c0937b1f admin admin] PATH_INFO: `/v3/projects/ce5bba490b034631900a08a305fb41a9` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:07.967139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35497de4-abad-429e-b583-0d17c0937b1f admin admin] RBAC: Authorizing `identity:delete_project(project_id=ce5bba490b034631900a08a305fb41a9)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:07.971313 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35497de4-abad-429e-b583-0d17c0937b1f admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:08.101825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-35497de4-abad-429e-b583-0d17c0937b1f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ce5bba490b034631900a08a305fb41a9', 'request_id': 'req-35497de4-abad-429e-b583-0d17c0937b1f'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:08.103594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:08.111722 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-35497de4-abad-429e-b583-0d17c0937b1f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ce5bba490b034631900a08a305fb41a9', 'request_id': 'req-35497de4-abad-429e-b583-0d17c0937b1f'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:08.121690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-35497de4-abad-429e-b583-0d17c0937b1f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ce5bba490b034631900a08a305fb41a9', 'request_id': 'req-35497de4-abad-429e-b583-0d17c0937b1f'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:08.145398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-35497de4-abad-429e-b583-0d17c0937b1f admin admin] The token cache is being invalidate because project ce5bba490b034631900a08a305fb41a9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:08.145771 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-35497de4-abad-429e-b583-0d17c0937b1f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:08.147319 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-35497de4-abad-429e-b583-0d17c0937b1f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:08.150596 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 982/1951] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:07 2022] DELETE /identity/v3/projects/ce5bba490b034631900a08a305fb41a9 => generated 0 bytes in 498 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:08.155486 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee0ed96-c629-4e7b-9620-992c4469972d None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:08.155837 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee0ed96-c629-4e7b-9620-992c4469972d None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:08.156181 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee0ed96-c629-4e7b-9620-992c4469972d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:08.179426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-2ee0ed96-c629-4e7b-9620-992c4469972d None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:08.180627 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-2ee0ed96-c629-4e7b-9620-992c4469972d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:08.252475 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 970/1952] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:08.260787 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114782ac-4f91-4df2-afce-7ecd97685b3c None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:08.261078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114782ac-4f91-4df2-afce-7ecd97685b3c None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:08.261354 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114782ac-4f91-4df2-afce-7ecd97685b3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:08.281972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-114782ac-4f91-4df2-afce-7ecd97685b3c None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:08.283334 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-114782ac-4f91-4df2-afce-7ecd97685b3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:08.291833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 983/1953] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:08.296266 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee0ed96-c629-4e7b-9620-992c4469972d None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:08.306429 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee0ed96-c629-4e7b-9620-992c4469972d None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:08.311224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d0e8f6-21b0-4501-905d-0534c5213fa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:08.311828 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d0e8f6-21b0-4501-905d-0534c5213fa5 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:08.312105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d0e8f6-21b0-4501-905d-0534c5213fa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:08.312378 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d0e8f6-21b0-4501-905d-0534c5213fa5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:08.313022 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d0e8f6-21b0-4501-905d-0534c5213fa5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:58:08.313325 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d0e8f6-21b0-4501-905d-0534c5213fa5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:08.314945 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d0e8f6-21b0-4501-905d-0534c5213fa5 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:08.327297 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 971/1954] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:58:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:58:08.331787 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114782ac-4f91-4df2-afce-7ecd97685b3c None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:08.349061 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114782ac-4f91-4df2-afce-7ecd97685b3c None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:08.349061 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c511bedc-8c69-429d-80a8-dcc712e737fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:08.349061 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c511bedc-8c69-429d-80a8-dcc712e737fb admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:08.349061 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c511bedc-8c69-429d-80a8-dcc712e737fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:08.349660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c511bedc-8c69-429d-80a8-dcc712e737fb admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:08.349660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c511bedc-8c69-429d-80a8-dcc712e737fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:08.350600 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c511bedc-8c69-429d-80a8-dcc712e737fb admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:08.393662 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 984/1955] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:08 2022] POST /identity/v3/projects => generated 377 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:58:08.395259 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d0e8f6-21b0-4501-905d-0534c5213fa5 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:08.406704 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d0e8f6-21b0-4501-905d-0534c5213fa5 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:08.474834 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc89be3-cfd0-4ecb-8591-4b32ba8709ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:08.475752 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc89be3-cfd0-4ecb-8591-4b32ba8709ba admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:08.476046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc89be3-cfd0-4ecb-8591-4b32ba8709ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:08.476323 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc89be3-cfd0-4ecb-8591-4b32ba8709ba admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:08.477149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc89be3-cfd0-4ecb-8591-4b32ba8709ba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:08.479336 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc89be3-cfd0-4ecb-8591-4b32ba8709ba admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:08.517881 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 972/1956] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:08 2022] POST /identity/v3/users => generated 289 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:08.522961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c511bedc-8c69-429d-80a8-dcc712e737fb admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:08.542820 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c511bedc-8c69-429d-80a8-dcc712e737fb admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:08.548498 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ee4fd1-fc3f-4649-8381-de008de93863 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:08.549218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ee4fd1-fc3f-4649-8381-de008de93863 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:08.549707 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ee4fd1-fc3f-4649-8381-de008de93863 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:08.550067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ee4fd1-fc3f-4649-8381-de008de93863 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:08.550893 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ee4fd1-fc3f-4649-8381-de008de93863 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:08.553136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ee4fd1-fc3f-4649-8381-de008de93863 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:08.563709 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 985/1957] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:08 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:08.568312 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc89be3-cfd0-4ecb-8591-4b32ba8709ba admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:08.579393 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc89be3-cfd0-4ecb-8591-4b32ba8709ba admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:08.584168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64357f12-2c53-4b69-8b2d-022d162493a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:08.584783 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64357f12-2c53-4b69-8b2d-022d162493a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:08.585055 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64357f12-2c53-4b69-8b2d-022d162493a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:08.585330 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64357f12-2c53-4b69-8b2d-022d162493a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:08.585971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64357f12-2c53-4b69-8b2d-022d162493a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:08.587583 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64357f12-2c53-4b69-8b2d-022d162493a1 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:08.599339 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 973/1958] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:08 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:08.603560 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ee4fd1-fc3f-4649-8381-de008de93863 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:08.615013 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ee4fd1-fc3f-4649-8381-de008de93863 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:08.620523 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9026b1fa-ed7e-4bec-95ff-df7818ac3096 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:08.621194 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9026b1fa-ed7e-4bec-95ff-df7818ac3096 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:08.621633 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9026b1fa-ed7e-4bec-95ff-df7818ac3096 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:08.621765 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9026b1fa-ed7e-4bec-95ff-df7818ac3096 admin admin] PATH_INFO: `/v3/projects/0a14df2380ab499fa25e831d9ca7585b/users/c8456ba7deaa4309839edaca6641ba05/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:08.640909 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9026b1fa-ed7e-4bec-95ff-df7818ac3096 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a14df2380ab499fa25e831d9ca7585b, user_id=c8456ba7deaa4309839edaca6641ba05, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:08.643950 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9026b1fa-ed7e-4bec-95ff-df7818ac3096 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:08.658102 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 986/1959] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:08 2022] PUT /identity/v3/projects/0a14df2380ab499fa25e831d9ca7585b/users/c8456ba7deaa4309839edaca6641ba05/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:08.676108 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d25821-fd68-450f-b7b8-6bce0a304200 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:08.676423 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d25821-fd68-450f-b7b8-6bce0a304200 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:08.676700 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d25821-fd68-450f-b7b8-6bce0a304200 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:08.710116 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-08d25821-fd68-450f-b7b8-6bce0a304200 None None] MFA Rules not processed for user `c8456ba7deaa4309839edaca6641ba05`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:08.711373 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-08d25821-fd68-450f-b7b8-6bce0a304200 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:08.747111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:08.783333 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 974/1960] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:08 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:08.788568 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9026b1fa-ed7e-4bec-95ff-df7818ac3096 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:08.857747 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9026b1fa-ed7e-4bec-95ff-df7818ac3096 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:08.862712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2c3d46-140c-4f60-a34a-e4a5cc242156 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:08.863328 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2c3d46-140c-4f60-a34a-e4a5cc242156 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:08.863602 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2c3d46-140c-4f60-a34a-e4a5cc242156 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:08.863871 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2c3d46-140c-4f60-a34a-e4a5cc242156 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:08.865123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2c3d46-140c-4f60-a34a-e4a5cc242156 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:08.867024 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2c3d46-140c-4f60-a34a-e4a5cc242156 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:08.899087 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 987/1961] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:08 2022] POST /identity/v3/projects => generated 379 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:08.903372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d25821-fd68-450f-b7b8-6bce0a304200 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:08.915530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d25821-fd68-450f-b7b8-6bce0a304200 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:08.992334 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1735724a-d852-4b27-a67b-987c8739899c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:08.993388 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1735724a-d852-4b27-a67b-987c8739899c admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:08.993883 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1735724a-d852-4b27-a67b-987c8739899c admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:08.994368 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1735724a-d852-4b27-a67b-987c8739899c admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:08.995602 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1735724a-d852-4b27-a67b-987c8739899c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:08.999465 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1735724a-d852-4b27-a67b-987c8739899c admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:09.033816 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 975/1962] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:08 2022] POST /identity/v3/users => generated 290 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:58:09.035294 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2c3d46-140c-4f60-a34a-e4a5cc242156 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:09.046579 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2c3d46-140c-4f60-a34a-e4a5cc242156 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:09.051407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2511f55d-4371-4bad-8b7c-aeab84cb750b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:09.052051 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2511f55d-4371-4bad-8b7c-aeab84cb750b admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:09.052333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2511f55d-4371-4bad-8b7c-aeab84cb750b admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:09.052626 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2511f55d-4371-4bad-8b7c-aeab84cb750b admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:09.053292 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2511f55d-4371-4bad-8b7c-aeab84cb750b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:09.054931 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2511f55d-4371-4bad-8b7c-aeab84cb750b admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:09.069191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 988/1963] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:09 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:58:09.074707 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1735724a-d852-4b27-a67b-987c8739899c admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:09.090890 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1735724a-d852-4b27-a67b-987c8739899c admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:09.095738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336db878-8346-450a-857b-77dbba6dbcbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:09.096348 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336db878-8346-450a-857b-77dbba6dbcbf admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:09.096626 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336db878-8346-450a-857b-77dbba6dbcbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:09.096896 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336db878-8346-450a-857b-77dbba6dbcbf admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:09.097544 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336db878-8346-450a-857b-77dbba6dbcbf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:09.099289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336db878-8346-450a-857b-77dbba6dbcbf admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:09.110996 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 976/1964] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:09 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:09.115595 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2511f55d-4371-4bad-8b7c-aeab84cb750b admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:09.125554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2511f55d-4371-4bad-8b7c-aeab84cb750b admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:09.130157 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebeac5fe-212d-4e12-9cc7-0ffaa758a9be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:09.130847 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebeac5fe-212d-4e12-9cc7-0ffaa758a9be admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:09.131128 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebeac5fe-212d-4e12-9cc7-0ffaa758a9be admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:09.131399 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebeac5fe-212d-4e12-9cc7-0ffaa758a9be admin admin] PATH_INFO: `/v3/projects/8c94659be8a34c6ebd6479f3244df4a0/users/b4a3987a183d46ceb4d34d51b16d37fc/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:09.148277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebeac5fe-212d-4e12-9cc7-0ffaa758a9be admin admin] RBAC: Authorizing `identity:create_grant(project_id=8c94659be8a34c6ebd6479f3244df4a0, user_id=b4a3987a183d46ceb4d34d51b16d37fc, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:09.151235 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebeac5fe-212d-4e12-9cc7-0ffaa758a9be admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:09.166285 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 989/1965] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:09 2022] PUT /identity/v3/projects/8c94659be8a34c6ebd6479f3244df4a0/users/b4a3987a183d46ceb4d34d51b16d37fc/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:09.177209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d23f767-015d-4556-a738-2ce22af8dbae None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:09.177494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d23f767-015d-4556-a738-2ce22af8dbae None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:09.177768 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d23f767-015d-4556-a738-2ce22af8dbae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:09.213335 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-1d23f767-015d-4556-a738-2ce22af8dbae None None] MFA Rules not processed for user `b4a3987a183d46ceb4d34d51b16d37fc`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:09.215488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-1d23f767-015d-4556-a738-2ce22af8dbae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:09.306752 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 977/1966] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:09 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:09.311241 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebeac5fe-212d-4e12-9cc7-0ffaa758a9be admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:09.342314 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:DELETE=3,keystone:SELECT=92,keystone:INSERT=5 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:09.384295 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebeac5fe-212d-4e12-9cc7-0ffaa758a9be admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:09.388989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b447c774-0eaf-4c03-b022-e3e485391ebe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:09.389625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b447c774-0eaf-4c03-b022-e3e485391ebe admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:09.389901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b447c774-0eaf-4c03-b022-e3e485391ebe admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:09.390287 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b447c774-0eaf-4c03-b022-e3e485391ebe admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:09.390839 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b447c774-0eaf-4c03-b022-e3e485391ebe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:09.392730 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b447c774-0eaf-4c03-b022-e3e485391ebe admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:09.421495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 990/1967] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:09 2022] POST /identity/v3/projects => generated 379 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:09.425566 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d23f767-015d-4556-a738-2ce22af8dbae None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:09.436963 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d23f767-015d-4556-a738-2ce22af8dbae None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:09.481791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:09.499031 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3483fb06-c0dd-4e79-82ad-4a8bfa9d3967 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:09.499652 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3483fb06-c0dd-4e79-82ad-4a8bfa9d3967 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:09.499960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3483fb06-c0dd-4e79-82ad-4a8bfa9d3967 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:09.500236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3483fb06-c0dd-4e79-82ad-4a8bfa9d3967 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:09.500878 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3483fb06-c0dd-4e79-82ad-4a8bfa9d3967 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:09.503879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3483fb06-c0dd-4e79-82ad-4a8bfa9d3967 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:09.532918 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 978/1968] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:09 2022] POST /identity/v3/users => generated 296 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:09.537969 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b447c774-0eaf-4c03-b022-e3e485391ebe admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:09.549885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b447c774-0eaf-4c03-b022-e3e485391ebe admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:09.555116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cd398d-9eaf-4e15-8b98-cae5c2f6e179 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:09.555734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cd398d-9eaf-4e15-8b98-cae5c2f6e179 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:09.557573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cd398d-9eaf-4e15-8b98-cae5c2f6e179 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:09.557870 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cd398d-9eaf-4e15-8b98-cae5c2f6e179 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:09.558678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6cd398d-9eaf-4e15-8b98-cae5c2f6e179 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:09.560301 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6cd398d-9eaf-4e15-8b98-cae5c2f6e179 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:09.580447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 991/1969] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:09 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:09.587821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3483fb06-c0dd-4e79-82ad-4a8bfa9d3967 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:09.606149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3483fb06-c0dd-4e79-82ad-4a8bfa9d3967 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:09.610089 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27aebf6-7f65-4434-8be9-b9e89dea424e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:09.610972 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27aebf6-7f65-4434-8be9-b9e89dea424e admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:09.611258 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27aebf6-7f65-4434-8be9-b9e89dea424e admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:09.611983 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27aebf6-7f65-4434-8be9-b9e89dea424e admin admin] PATH_INFO: `/v3/projects/cfd8ac6e394049d79bfa3e24a525f293/users/aac35b6e0efa4450ab5b15f60192439a/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:09.627619 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c27aebf6-7f65-4434-8be9-b9e89dea424e admin admin] RBAC: Authorizing `identity:create_grant(project_id=cfd8ac6e394049d79bfa3e24a525f293, user_id=aac35b6e0efa4450ab5b15f60192439a, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:09.630836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c27aebf6-7f65-4434-8be9-b9e89dea424e admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:09.647754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 979/1970] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:09 2022] PUT /identity/v3/projects/cfd8ac6e394049d79bfa3e24a525f293/users/aac35b6e0efa4450ab5b15f60192439a/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:09.651646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cd398d-9eaf-4e15-8b98-cae5c2f6e179 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:09.663606 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cd398d-9eaf-4e15-8b98-cae5c2f6e179 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:09.744962 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21491e8a-4701-4b0d-bea8-c58b16d499a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:09.745628 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21491e8a-4701-4b0d-bea8-c58b16d499a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:09.745962 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21491e8a-4701-4b0d-bea8-c58b16d499a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:09.746347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21491e8a-4701-4b0d-bea8-c58b16d499a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:09.747314 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21491e8a-4701-4b0d-bea8-c58b16d499a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:09.749574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21491e8a-4701-4b0d-bea8-c58b16d499a3 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:09.763722 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 992/1971] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:09 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:09.764925 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27aebf6-7f65-4434-8be9-b9e89dea424e admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:09.775489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27aebf6-7f65-4434-8be9-b9e89dea424e admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:09.780311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e834d1-06e6-4824-9cd6-e173de657e01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:09.780988 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e834d1-06e6-4824-9cd6-e173de657e01 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:09.781265 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e834d1-06e6-4824-9cd6-e173de657e01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:09.781538 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e834d1-06e6-4824-9cd6-e173de657e01 admin admin] PATH_INFO: `/v3/projects/cfd8ac6e394049d79bfa3e24a525f293/users/aac35b6e0efa4450ab5b15f60192439a/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:09.783549 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e834d1-06e6-4824-9cd6-e173de657e01 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cfd8ac6e394049d79bfa3e24a525f293, user_id=aac35b6e0efa4450ab5b15f60192439a, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:09.786318 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e834d1-06e6-4824-9cd6-e173de657e01 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:09.797807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 980/1972] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:09 2022] PUT /identity/v3/projects/cfd8ac6e394049d79bfa3e24a525f293/users/aac35b6e0efa4450ab5b15f60192439a/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:09.810745 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd46c3d-3eb2-4de5-a206-d1450deabb3b None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:09.811289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd46c3d-3eb2-4de5-a206-d1450deabb3b None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:09.811577 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd46c3d-3eb2-4de5-a206-d1450deabb3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:09.844625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-cfd46c3d-3eb2-4de5-a206-d1450deabb3b None None] MFA Rules not processed for user `aac35b6e0efa4450ab5b15f60192439a`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:09.845847 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-cfd46c3d-3eb2-4de5-a206-d1450deabb3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:09.915809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 993/1973] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:09 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:09.926845 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e834d1-06e6-4824-9cd6-e173de657e01 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:09.929076 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-b3e834d1-06e6-4824-9cd6-e173de657e01 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:10.000810 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e834d1-06e6-4824-9cd6-e173de657e01 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:10.004529 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66780035-8d49-4300-be29-74262e633bf2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:10.005797 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66780035-8d49-4300-be29-74262e633bf2 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:10.006268 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66780035-8d49-4300-be29-74262e633bf2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:10.006763 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66780035-8d49-4300-be29-74262e633bf2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:10.019621 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66780035-8d49-4300-be29-74262e633bf2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:10.021455 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66780035-8d49-4300-be29-74262e633bf2 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:10.031954 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 981/1974] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:09 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:10.084106 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd46c3d-3eb2-4de5-a206-d1450deabb3b None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:10.086071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-cfd46c3d-3eb2-4de5-a206-d1450deabb3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:10.102212 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd46c3d-3eb2-4de5-a206-d1450deabb3b None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:10.106537 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699c1a37-b865-419d-a466-e8108b6157b4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:10.107167 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699c1a37-b865-419d-a466-e8108b6157b4 service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:10.107449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699c1a37-b865-419d-a466-e8108b6157b4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:10.107722 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699c1a37-b865-419d-a466-e8108b6157b4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:10.132444 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:10.190828 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699c1a37-b865-419d-a466-e8108b6157b4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:10.192949 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699c1a37-b865-419d-a466-e8108b6157b4 service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:10.203967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 994/1975] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:10 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:12.105334 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66780035-8d49-4300-be29-74262e633bf2 service cinder] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:12.109003 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-66780035-8d49-4300-be29-74262e633bf2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:12.217522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66780035-8d49-4300-be29-74262e633bf2 service cinder] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:12.224469 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d01119-a018-4b33-a6b5-da9e2cf4028a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:12.225449 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d01119-a018-4b33-a6b5-da9e2cf4028a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:12.226018 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d01119-a018-4b33-a6b5-da9e2cf4028a admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:12.227072 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d01119-a018-4b33-a6b5-da9e2cf4028a admin admin] PATH_INFO: `/v3/users/50fb1afce26a47a9848da001454a23b0` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:12.228674 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d01119-a018-4b33-a6b5-da9e2cf4028a admin admin] RBAC: Authorizing `identity:delete_user(user_id=50fb1afce26a47a9848da001454a23b0)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:12.230648 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d01119-a018-4b33-a6b5-da9e2cf4028a admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:12.301794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=90,keystone:INSERT=5,keystone:DELETE=5 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:12.317126 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-86d01119-a018-4b33-a6b5-da9e2cf4028a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '50fb1afce26a47a9848da001454a23b0'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:12.332905 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-86d01119-a018-4b33-a6b5-da9e2cf4028a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '50fb1afce26a47a9848da001454a23b0'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:12.333893 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-86d01119-a018-4b33-a6b5-da9e2cf4028a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '50fb1afce26a47a9848da001454a23b0'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:12.335235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-86d01119-a018-4b33-a6b5-da9e2cf4028a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '50fb1afce26a47a9848da001454a23b0'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:12.350702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-86d01119-a018-4b33-a6b5-da9e2cf4028a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '50fb1afce26a47a9848da001454a23b0'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:12.370179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 982/1976] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:12 2022] DELETE /identity/v3/users/50fb1afce26a47a9848da001454a23b0 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:12.375557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3578778e-467f-490d-bf02-c75bfcbdaeb0 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:12.375557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3578778e-467f-490d-bf02-c75bfcbdaeb0 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:12.376788 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3578778e-467f-490d-bf02-c75bfcbdaeb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:12.402276 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-3578778e-467f-490d-bf02-c75bfcbdaeb0 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:12.403826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3578778e-467f-490d-bf02-c75bfcbdaeb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:12.488837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 995/1977] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:12.499238 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d01119-a018-4b33-a6b5-da9e2cf4028a admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:12.502912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-86d01119-a018-4b33-a6b5-da9e2cf4028a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:12.574805 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d01119-a018-4b33-a6b5-da9e2cf4028a admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:12.578735 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c076dc-80f0-4f18-8e8b-03aa046de436 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:12.579511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c076dc-80f0-4f18-8e8b-03aa046de436 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:12.579862 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c076dc-80f0-4f18-8e8b-03aa046de436 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:12.580144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c076dc-80f0-4f18-8e8b-03aa046de436 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:12.590019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c076dc-80f0-4f18-8e8b-03aa046de436 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:12.591513 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c076dc-80f0-4f18-8e8b-03aa046de436 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:12.601674 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 983/1978] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:12.940286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3578778e-467f-490d-bf02-c75bfcbdaeb0 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:12.941957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3578778e-467f-490d-bf02-c75bfcbdaeb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:12.965179 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3578778e-467f-490d-bf02-c75bfcbdaeb0 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:12.971165 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd37b716-50a6-46d3-b266-7f84c0e2c3d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:12.971987 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd37b716-50a6-46d3-b266-7f84c0e2c3d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:12.972292 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd37b716-50a6-46d3-b266-7f84c0e2c3d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:12.972614 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd37b716-50a6-46d3-b266-7f84c0e2c3d5 admin admin] PATH_INFO: `/v3/projects/d40411f11d0740b3946a1f8fdded43f6` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:12.975293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd37b716-50a6-46d3-b266-7f84c0e2c3d5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d40411f11d0740b3946a1f8fdded43f6)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:12.977709 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd37b716-50a6-46d3-b266-7f84c0e2c3d5 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:13.046074 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-dd37b716-50a6-46d3-b266-7f84c0e2c3d5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd40411f11d0740b3946a1f8fdded43f6', 'request_id': 'req-dd37b716-50a6-46d3-b266-7f84c0e2c3d5'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:13.053333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-dd37b716-50a6-46d3-b266-7f84c0e2c3d5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd40411f11d0740b3946a1f8fdded43f6', 'request_id': 'req-dd37b716-50a6-46d3-b266-7f84c0e2c3d5'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:13.065421 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-dd37b716-50a6-46d3-b266-7f84c0e2c3d5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd40411f11d0740b3946a1f8fdded43f6', 'request_id': 'req-dd37b716-50a6-46d3-b266-7f84c0e2c3d5'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:13.081537 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-dd37b716-50a6-46d3-b266-7f84c0e2c3d5 admin admin] The token cache is being invalidate because project d40411f11d0740b3946a1f8fdded43f6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:13.081857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-dd37b716-50a6-46d3-b266-7f84c0e2c3d5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:13.082591 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-dd37b716-50a6-46d3-b266-7f84c0e2c3d5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:13.085217 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 996/1979] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:12 2022] DELETE /identity/v3/projects/d40411f11d0740b3946a1f8fdded43f6 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:13.089851 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c076dc-80f0-4f18-8e8b-03aa046de436 service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:13.104866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c076dc-80f0-4f18-8e8b-03aa046de436 service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:13.194332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e9c7c0-0600-41d5-925a-05f303b38a53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:13.195050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e9c7c0-0600-41d5-925a-05f303b38a53 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:13.195276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e9c7c0-0600-41d5-925a-05f303b38a53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:13.195548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e9c7c0-0600-41d5-925a-05f303b38a53 admin admin] PATH_INFO: `/v3/users/7db453a24c5c48fc9845f5a544e12b49` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:13.197025 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e9c7c0-0600-41d5-925a-05f303b38a53 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7db453a24c5c48fc9845f5a544e12b49)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:13.199336 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e9c7c0-0600-41d5-925a-05f303b38a53 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:13.254224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-f9e9c7c0-0600-41d5-925a-05f303b38a53 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7db453a24c5c48fc9845f5a544e12b49'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:13.264111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-f9e9c7c0-0600-41d5-925a-05f303b38a53 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7db453a24c5c48fc9845f5a544e12b49'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:13.264848 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-f9e9c7c0-0600-41d5-925a-05f303b38a53 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7db453a24c5c48fc9845f5a544e12b49'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:13.265725 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-f9e9c7c0-0600-41d5-925a-05f303b38a53 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7db453a24c5c48fc9845f5a544e12b49'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:13.272643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:DELETE=13,keystone:SELECT=85,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:13.275365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-f9e9c7c0-0600-41d5-925a-05f303b38a53 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7db453a24c5c48fc9845f5a544e12b49'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:13.312339 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 984/1980] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:13 2022] DELETE /identity/v3/users/7db453a24c5c48fc9845f5a544e12b49 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:13.419164 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd37b716-50a6-46d3-b266-7f84c0e2c3d5 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:13.433884 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-dd37b716-50a6-46d3-b266-7f84c0e2c3d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:13.453883 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd37b716-50a6-46d3-b266-7f84c0e2c3d5 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:13.518586 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:13.572809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5cdeab-767a-4e24-9b4d-afb92729b97f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:13.573470 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5cdeab-767a-4e24-9b4d-afb92729b97f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:13.573753 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5cdeab-767a-4e24-9b4d-afb92729b97f admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:13.574029 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5cdeab-767a-4e24-9b4d-afb92729b97f admin admin] PATH_INFO: `/v3/users/c8456ba7deaa4309839edaca6641ba05` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:13.575373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5cdeab-767a-4e24-9b4d-afb92729b97f admin admin] RBAC: Authorizing `identity:delete_user(user_id=c8456ba7deaa4309839edaca6641ba05)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:13.577401 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5cdeab-767a-4e24-9b4d-afb92729b97f admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:13.645061 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-3d5cdeab-767a-4e24-9b4d-afb92729b97f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c8456ba7deaa4309839edaca6641ba05'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:13.654917 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-3d5cdeab-767a-4e24-9b4d-afb92729b97f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c8456ba7deaa4309839edaca6641ba05'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:13.678608 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-3d5cdeab-767a-4e24-9b4d-afb92729b97f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c8456ba7deaa4309839edaca6641ba05'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:13.680607 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-3d5cdeab-767a-4e24-9b4d-afb92729b97f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c8456ba7deaa4309839edaca6641ba05'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:13.694546 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-3d5cdeab-767a-4e24-9b4d-afb92729b97f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c8456ba7deaa4309839edaca6641ba05'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:13.703561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 997/1981] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:13 2022] DELETE /identity/v3/users/c8456ba7deaa4309839edaca6641ba05 => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:13.707791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934fb90e-77a1-4e1c-95bc-c2d8c3617532 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:13.708105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934fb90e-77a1-4e1c-95bc-c2d8c3617532 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:13.708936 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934fb90e-77a1-4e1c-95bc-c2d8c3617532 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:13.749452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-934fb90e-77a1-4e1c-95bc-c2d8c3617532 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:13.751290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-934fb90e-77a1-4e1c-95bc-c2d8c3617532 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:13.870594 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 985/1982] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:13.885839 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5cdeab-767a-4e24-9b4d-afb92729b97f admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:13.890017 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3d5cdeab-767a-4e24-9b4d-afb92729b97f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:13.953844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934fb90e-77a1-4e1c-95bc-c2d8c3617532 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:13.955685 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-934fb90e-77a1-4e1c-95bc-c2d8c3617532 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:13.969109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934fb90e-77a1-4e1c-95bc-c2d8c3617532 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:13.985074 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c958ea7-c375-4eae-813c-3667915668dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:13.985770 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c958ea7-c375-4eae-813c-3667915668dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:13.986054 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c958ea7-c375-4eae-813c-3667915668dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:13.986342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c958ea7-c375-4eae-813c-3667915668dc admin admin] PATH_INFO: `/v3/projects/6b3085ac38454731ae03e679739e1503` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:14.006111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c958ea7-c375-4eae-813c-3667915668dc admin admin] RBAC: Authorizing `identity:delete_project(project_id=6b3085ac38454731ae03e679739e1503)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:14.006111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c958ea7-c375-4eae-813c-3667915668dc admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:14.018305 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5cdeab-767a-4e24-9b4d-afb92729b97f admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:14.033752 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af994c3-febd-41d3-9c0a-2894c29d4791 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:14.034277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af994c3-febd-41d3-9c0a-2894c29d4791 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:14.034583 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af994c3-febd-41d3-9c0a-2894c29d4791 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:14.034853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af994c3-febd-41d3-9c0a-2894c29d4791 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:14.046546 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af994c3-febd-41d3-9c0a-2894c29d4791 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:14.048746 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af994c3-febd-41d3-9c0a-2894c29d4791 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:14.066965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 998/1983] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:14.070361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-7c958ea7-c375-4eae-813c-3667915668dc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6b3085ac38454731ae03e679739e1503', 'request_id': 'req-7c958ea7-c375-4eae-813c-3667915668dc'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:14.077606 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-7c958ea7-c375-4eae-813c-3667915668dc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6b3085ac38454731ae03e679739e1503', 'request_id': 'req-7c958ea7-c375-4eae-813c-3667915668dc'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:14.087078 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-7c958ea7-c375-4eae-813c-3667915668dc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6b3085ac38454731ae03e679739e1503', 'request_id': 'req-7c958ea7-c375-4eae-813c-3667915668dc'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:14.099678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-7c958ea7-c375-4eae-813c-3667915668dc admin admin] The token cache is being invalidate because project 6b3085ac38454731ae03e679739e1503 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:14.099964 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-7c958ea7-c375-4eae-813c-3667915668dc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:14.100638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-7c958ea7-c375-4eae-813c-3667915668dc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:14.102138 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 986/1984] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:13 2022] DELETE /identity/v3/projects/6b3085ac38454731ae03e679739e1503 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:14.109783 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95118d5-f327-4a36-ab8d-eeab9e1500db None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:14.110251 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95118d5-f327-4a36-ab8d-eeab9e1500db None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:14.110739 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95118d5-f327-4a36-ab8d-eeab9e1500db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:14.136224 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-e95118d5-f327-4a36-ab8d-eeab9e1500db None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:14.137844 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-e95118d5-f327-4a36-ab8d-eeab9e1500db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:14.226662 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 999/1985] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:58:14.235392 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62097a3a-8ef3-42c8-a3c4-35cd0bb60437 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:14.235970 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62097a3a-8ef3-42c8-a3c4-35cd0bb60437 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:14.235970 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62097a3a-8ef3-42c8-a3c4-35cd0bb60437 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:14.259612 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-62097a3a-8ef3-42c8-a3c4-35cd0bb60437 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:14.260829 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-62097a3a-8ef3-42c8-a3c4-35cd0bb60437 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:14.270047 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 987/1986] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:14.274452 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95118d5-f327-4a36-ab8d-eeab9e1500db None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:14.281780 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:14.286932 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95118d5-f327-4a36-ab8d-eeab9e1500db None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:14.294131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13a76d4-e2e9-49f8-83e6-2d630431c6bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:14.294969 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13a76d4-e2e9-49f8-83e6-2d630431c6bc admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:14.295421 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13a76d4-e2e9-49f8-83e6-2d630431c6bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:14.295793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13a76d4-e2e9-49f8-83e6-2d630431c6bc admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:14.296549 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13a76d4-e2e9-49f8-83e6-2d630431c6bc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:58:14.297243 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13a76d4-e2e9-49f8-83e6-2d630431c6bc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:14.298988 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13a76d4-e2e9-49f8-83e6-2d630431c6bc admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:14.311783 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1000/1987] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:58:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:58:14.316324 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62097a3a-8ef3-42c8-a3c4-35cd0bb60437 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:14.327019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62097a3a-8ef3-42c8-a3c4-35cd0bb60437 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:14.331729 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84eec9f-0b81-4a0a-8a9e-7fabb7394b0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:14.332366 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84eec9f-0b81-4a0a-8a9e-7fabb7394b0f admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:14.332603 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84eec9f-0b81-4a0a-8a9e-7fabb7394b0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:14.332913 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84eec9f-0b81-4a0a-8a9e-7fabb7394b0f admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:14.332913 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84eec9f-0b81-4a0a-8a9e-7fabb7394b0f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:14.335702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84eec9f-0b81-4a0a-8a9e-7fabb7394b0f admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:14.371331 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 988/1988] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:14 2022] POST /identity/v3/projects => generated 367 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:58:14.372767 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13a76d4-e2e9-49f8-83e6-2d630431c6bc admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:14.385784 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13a76d4-e2e9-49f8-83e6-2d630431c6bc admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:14.388538 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84eec9f-0b81-4a0a-8a9e-7fabb7394b0f admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:14.391734 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d84eec9f-0b81-4a0a-8a9e-7fabb7394b0f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:14.445705 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=9,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:14.476645 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4edb4d-5fb4-4c81-b1d8-a31b0aaae249 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:14.477427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4edb4d-5fb4-4c81-b1d8-a31b0aaae249 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:14.477808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4edb4d-5fb4-4c81-b1d8-a31b0aaae249 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:14.478150 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4edb4d-5fb4-4c81-b1d8-a31b0aaae249 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:14.479022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d4edb4d-5fb4-4c81-b1d8-a31b0aaae249 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:14.481050 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d4edb4d-5fb4-4c81-b1d8-a31b0aaae249 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:14.485261 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84eec9f-0b81-4a0a-8a9e-7fabb7394b0f admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:14.490756 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed11d14b-ba23-420a-8f0d-7b67f032e1bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:14.491544 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed11d14b-ba23-420a-8f0d-7b67f032e1bd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:14.492078 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed11d14b-ba23-420a-8f0d-7b67f032e1bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:14.492449 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed11d14b-ba23-420a-8f0d-7b67f032e1bd admin admin] PATH_INFO: `/v3/projects/0a14df2380ab499fa25e831d9ca7585b` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:14.493707 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed11d14b-ba23-420a-8f0d-7b67f032e1bd admin admin] RBAC: Authorizing `identity:delete_project(project_id=0a14df2380ab499fa25e831d9ca7585b)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:14.496032 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed11d14b-ba23-420a-8f0d-7b67f032e1bd admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:14.515284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1001/1989] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:14 2022] POST /identity/v3/users => generated 284 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:14.518995 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4edb4d-5fb4-4c81-b1d8-a31b0aaae249 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:14.532000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4edb4d-5fb4-4c81-b1d8-a31b0aaae249 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:14.538806 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10a5598-d9da-4116-b2cd-9f0ece980085 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:14.539429 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10a5598-d9da-4116-b2cd-9f0ece980085 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:14.539707 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10a5598-d9da-4116-b2cd-9f0ece980085 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:14.539976 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10a5598-d9da-4116-b2cd-9f0ece980085 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:14.540627 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10a5598-d9da-4116-b2cd-9f0ece980085 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:14.542235 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10a5598-d9da-4116-b2cd-9f0ece980085 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:14.554739 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1002/1990] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:14 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:58:14.555637 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10a5598-d9da-4116-b2cd-9f0ece980085 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:14.557320 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-b10a5598-d9da-4116-b2cd-9f0ece980085 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:14.569947 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10a5598-d9da-4116-b2cd-9f0ece980085 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:14.579775 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-ed11d14b-ba23-420a-8f0d-7b67f032e1bd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0a14df2380ab499fa25e831d9ca7585b', 'request_id': 'req-ed11d14b-ba23-420a-8f0d-7b67f032e1bd'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:14.589378 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-ed11d14b-ba23-420a-8f0d-7b67f032e1bd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0a14df2380ab499fa25e831d9ca7585b', 'request_id': 'req-ed11d14b-ba23-420a-8f0d-7b67f032e1bd'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:14.600667 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-ed11d14b-ba23-420a-8f0d-7b67f032e1bd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0a14df2380ab499fa25e831d9ca7585b', 'request_id': 'req-ed11d14b-ba23-420a-8f0d-7b67f032e1bd'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:14.613778 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-ed11d14b-ba23-420a-8f0d-7b67f032e1bd admin admin] The token cache is being invalidate because project 0a14df2380ab499fa25e831d9ca7585b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:14.614071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-ed11d14b-ba23-420a-8f0d-7b67f032e1bd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:14.615343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-ed11d14b-ba23-420a-8f0d-7b67f032e1bd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:14.616933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 989/1991] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:14 2022] DELETE /identity/v3/projects/0a14df2380ab499fa25e831d9ca7585b => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:14.619584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed11d14b-ba23-420a-8f0d-7b67f032e1bd admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:14.621730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ed11d14b-ba23-420a-8f0d-7b67f032e1bd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:14.633234 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32d9ce1-e308-4ec1-bd4c-c5728669119c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:14.633977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32d9ce1-e308-4ec1-bd4c-c5728669119c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:14.634327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32d9ce1-e308-4ec1-bd4c-c5728669119c admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:14.634675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32d9ce1-e308-4ec1-bd4c-c5728669119c admin admin] PATH_INFO: `/v3/users/6d507dc6ddd04533a4c570714a5fa996` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:14.636044 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32d9ce1-e308-4ec1-bd4c-c5728669119c admin admin] RBAC: Authorizing `identity:delete_user(user_id=6d507dc6ddd04533a4c570714a5fa996)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:14.638303 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32d9ce1-e308-4ec1-bd4c-c5728669119c admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:14.692694 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a32d9ce1-e308-4ec1-bd4c-c5728669119c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6d507dc6ddd04533a4c570714a5fa996'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:14.696204 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed11d14b-ba23-420a-8f0d-7b67f032e1bd admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:14.700865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a32d9ce1-e308-4ec1-bd4c-c5728669119c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6d507dc6ddd04533a4c570714a5fa996'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:14.701657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5897f36-cf92-46c9-96b5-f4bda4e5ba8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:14.702257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5897f36-cf92-46c9-96b5-f4bda4e5ba8a admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:14.702567 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5897f36-cf92-46c9-96b5-f4bda4e5ba8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:14.702843 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5897f36-cf92-46c9-96b5-f4bda4e5ba8a admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:14.703490 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5897f36-cf92-46c9-96b5-f4bda4e5ba8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:14.705086 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5897f36-cf92-46c9-96b5-f4bda4e5ba8a admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:14.714900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a32d9ce1-e308-4ec1-bd4c-c5728669119c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6d507dc6ddd04533a4c570714a5fa996'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:14.715796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a32d9ce1-e308-4ec1-bd4c-c5728669119c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6d507dc6ddd04533a4c570714a5fa996'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:14.715929 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 990/1992] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:14 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:14.717994 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:14.722718 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5897f36-cf92-46c9-96b5-f4bda4e5ba8a admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:14.724676 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c5897f36-cf92-46c9-96b5-f4bda4e5ba8a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:14.730583 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a32d9ce1-e308-4ec1-bd4c-c5728669119c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6d507dc6ddd04533a4c570714a5fa996'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:14.738264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1003/1993] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:14 2022] DELETE /identity/v3/users/6d507dc6ddd04533a4c570714a5fa996 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:14.739278 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32d9ce1-e308-4ec1-bd4c-c5728669119c admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:14.739407 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5897f36-cf92-46c9-96b5-f4bda4e5ba8a admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:14.740733 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a32d9ce1-e308-4ec1-bd4c-c5728669119c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:14.799768 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8becd89-aee0-44c9-a182-50bde0dec649 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:14.800565 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8becd89-aee0-44c9-a182-50bde0dec649 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:14.800943 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8becd89-aee0-44c9-a182-50bde0dec649 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:14.801315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8becd89-aee0-44c9-a182-50bde0dec649 admin admin] PATH_INFO: `/v3/users/b4a3987a183d46ceb4d34d51b16d37fc` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:14.803239 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8becd89-aee0-44c9-a182-50bde0dec649 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b4a3987a183d46ceb4d34d51b16d37fc)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:14.805328 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8becd89-aee0-44c9-a182-50bde0dec649 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:14.807376 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:14.812029 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32d9ce1-e308-4ec1-bd4c-c5728669119c admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:14.816833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e278b2a-1a20-411e-94f1-ed913709e666 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:14.817543 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e278b2a-1a20-411e-94f1-ed913709e666 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:14.817822 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e278b2a-1a20-411e-94f1-ed913709e666 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:14.818091 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e278b2a-1a20-411e-94f1-ed913709e666 admin admin] PATH_INFO: `/v3/projects/c6d771c1e2fe4b1b99982301423f0f13/users/9c54c0c5f339483ab56d52bab2f63a17/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:14.835024 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e278b2a-1a20-411e-94f1-ed913709e666 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6d771c1e2fe4b1b99982301423f0f13, user_id=9c54c0c5f339483ab56d52bab2f63a17, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:14.838447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e278b2a-1a20-411e-94f1-ed913709e666 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:14.855183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1004/1994] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:14 2022] PUT /identity/v3/projects/c6d771c1e2fe4b1b99982301423f0f13/users/9c54c0c5f339483ab56d52bab2f63a17/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:14.856778 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832bc023-6d1c-45df-bac2-8089112534fa None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:14.857032 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832bc023-6d1c-45df-bac2-8089112534fa None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:14.857303 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832bc023-6d1c-45df-bac2-8089112534fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:14.874919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c8becd89-aee0-44c9-a182-50bde0dec649 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b4a3987a183d46ceb4d34d51b16d37fc'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:14.885798 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-832bc023-6d1c-45df-bac2-8089112534fa None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:14.887045 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-832bc023-6d1c-45df-bac2-8089112534fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:14.892650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c8becd89-aee0-44c9-a182-50bde0dec649 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b4a3987a183d46ceb4d34d51b16d37fc'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:14.893905 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c8becd89-aee0-44c9-a182-50bde0dec649 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b4a3987a183d46ceb4d34d51b16d37fc'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:14.894785 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c8becd89-aee0-44c9-a182-50bde0dec649 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b4a3987a183d46ceb4d34d51b16d37fc'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:14.903522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c8becd89-aee0-44c9-a182-50bde0dec649 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b4a3987a183d46ceb4d34d51b16d37fc'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:14.922610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 991/1995] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:14 2022] DELETE /identity/v3/users/b4a3987a183d46ceb4d34d51b16d37fc => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:14.924012 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c254a23-9bbe-44dd-84fe-f0ded9aa6729 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:14.925051 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c254a23-9bbe-44dd-84fe-f0ded9aa6729 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:14.925484 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c254a23-9bbe-44dd-84fe-f0ded9aa6729 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:14.989054 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-0c254a23-9bbe-44dd-84fe-f0ded9aa6729 None None] MFA Rules not processed for user `9c54c0c5f339483ab56d52bab2f63a17`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:14.991425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-0c254a23-9bbe-44dd-84fe-f0ded9aa6729 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:15.002507 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1005/1996] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:15.024007 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832bc023-6d1c-45df-bac2-8089112534fa None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:15.029514 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-832bc023-6d1c-45df-bac2-8089112534fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:15.124481 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 992/1997] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:14 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:15.131151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c254a23-9bbe-44dd-84fe-f0ded9aa6729 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:15.134447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-0c254a23-9bbe-44dd-84fe-f0ded9aa6729 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:15.153564 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832bc023-6d1c-45df-bac2-8089112534fa None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:15.192245 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:15.222084 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676d36c0-51a9-489c-b0bf-6091d1a22866 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:15.222735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676d36c0-51a9-489c-b0bf-6091d1a22866 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:15.223028 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676d36c0-51a9-489c-b0bf-6091d1a22866 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:15.223291 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676d36c0-51a9-489c-b0bf-6091d1a22866 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:15.234340 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c254a23-9bbe-44dd-84fe-f0ded9aa6729 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:15.234895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676d36c0-51a9-489c-b0bf-6091d1a22866 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:15.236363 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676d36c0-51a9-489c-b0bf-6091d1a22866 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:15.240573 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a617e39b-d184-4e45-91c0-6eceb00d334a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:15.242135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a617e39b-d184-4e45-91c0-6eceb00d334a admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:15.242135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a617e39b-d184-4e45-91c0-6eceb00d334a admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:15.242690 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a617e39b-d184-4e45-91c0-6eceb00d334a admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:15.244776 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a617e39b-d184-4e45-91c0-6eceb00d334a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:15.254854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a617e39b-d184-4e45-91c0-6eceb00d334a admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:15.263822 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1006/1998] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:15.288587 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:15.309888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 993/1999] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:15 2022] POST /identity/v3/projects => generated 367 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:15.314209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676d36c0-51a9-489c-b0bf-6091d1a22866 service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:15.329845 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676d36c0-51a9-489c-b0bf-6091d1a22866 service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:15.379830 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a617e39b-d184-4e45-91c0-6eceb00d334a admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:15.382593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-a617e39b-d184-4e45-91c0-6eceb00d334a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:15.453327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0688de79-ee9c-4a56-af1c-144dbb74c074 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:15.454046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0688de79-ee9c-4a56-af1c-144dbb74c074 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:15.454516 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0688de79-ee9c-4a56-af1c-144dbb74c074 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:15.454752 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0688de79-ee9c-4a56-af1c-144dbb74c074 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:15.455545 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0688de79-ee9c-4a56-af1c-144dbb74c074 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:15.457563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0688de79-ee9c-4a56-af1c-144dbb74c074 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:15.487225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1007/2000] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:15 2022] POST /identity/v3/users => generated 290 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:58:15.491016 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0688de79-ee9c-4a56-af1c-144dbb74c074 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:15.496577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a617e39b-d184-4e45-91c0-6eceb00d334a admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:15.501134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0688de79-ee9c-4a56-af1c-144dbb74c074 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:15.503608 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58b2536-eb2d-402e-ba58-47fc2a8e70bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:15.504240 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58b2536-eb2d-402e-ba58-47fc2a8e70bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:15.504608 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58b2536-eb2d-402e-ba58-47fc2a8e70bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:15.505543 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58b2536-eb2d-402e-ba58-47fc2a8e70bf admin admin] PATH_INFO: `/v3/projects/8c94659be8a34c6ebd6479f3244df4a0` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:15.506816 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58b2536-eb2d-402e-ba58-47fc2a8e70bf admin admin] RBAC: Authorizing `identity:delete_project(project_id=8c94659be8a34c6ebd6479f3244df4a0)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:15.507037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb7fa2c-c1f4-4af3-9055-76b302f1a5b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:15.507711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb7fa2c-c1f4-4af3-9055-76b302f1a5b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:15.508084 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb7fa2c-c1f4-4af3-9055-76b302f1a5b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:15.508397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb7fa2c-c1f4-4af3-9055-76b302f1a5b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:15.509116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb7fa2c-c1f4-4af3-9055-76b302f1a5b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:15.509262 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58b2536-eb2d-402e-ba58-47fc2a8e70bf admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:15.510944 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb7fa2c-c1f4-4af3-9055-76b302f1a5b4 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:15.525463 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1008/2001] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:15 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:58:15.527478 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb7fa2c-c1f4-4af3-9055-76b302f1a5b4 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:15.538260 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb7fa2c-c1f4-4af3-9055-76b302f1a5b4 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:15.543570 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bba1ee4-6825-4055-ae1d-87da01aa636d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:15.544320 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bba1ee4-6825-4055-ae1d-87da01aa636d admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:15.544912 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bba1ee4-6825-4055-ae1d-87da01aa636d admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:15.545273 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bba1ee4-6825-4055-ae1d-87da01aa636d admin admin] PATH_INFO: `/v3/projects/525917f9910e403992fb40c1298a68a7/users/6d09a427b18a4be0b06a99eb9287e1a1/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:15.563787 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bba1ee4-6825-4055-ae1d-87da01aa636d admin admin] RBAC: Authorizing `identity:create_grant(project_id=525917f9910e403992fb40c1298a68a7, user_id=6d09a427b18a4be0b06a99eb9287e1a1, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:15.569603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bba1ee4-6825-4055-ae1d-87da01aa636d admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:15.585893 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-d58b2536-eb2d-402e-ba58-47fc2a8e70bf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8c94659be8a34c6ebd6479f3244df4a0', 'request_id': 'req-d58b2536-eb2d-402e-ba58-47fc2a8e70bf'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:15.588114 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1009/2002] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:15 2022] PUT /identity/v3/projects/525917f9910e403992fb40c1298a68a7/users/6d09a427b18a4be0b06a99eb9287e1a1/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:15.592854 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bba1ee4-6825-4055-ae1d-87da01aa636d admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:15.598212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-d58b2536-eb2d-402e-ba58-47fc2a8e70bf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8c94659be8a34c6ebd6479f3244df4a0', 'request_id': 'req-d58b2536-eb2d-402e-ba58-47fc2a8e70bf'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:15.616036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-d58b2536-eb2d-402e-ba58-47fc2a8e70bf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8c94659be8a34c6ebd6479f3244df4a0', 'request_id': 'req-d58b2536-eb2d-402e-ba58-47fc2a8e70bf'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:15.641333 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-d58b2536-eb2d-402e-ba58-47fc2a8e70bf admin admin] The token cache is being invalidate because project 8c94659be8a34c6ebd6479f3244df4a0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:15.641754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-d58b2536-eb2d-402e-ba58-47fc2a8e70bf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:15.642670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-d58b2536-eb2d-402e-ba58-47fc2a8e70bf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:15.644516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 994/2003] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:15 2022] DELETE /identity/v3/projects/8c94659be8a34c6ebd6479f3244df4a0 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:15.645796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58b2536-eb2d-402e-ba58-47fc2a8e70bf admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:15.648990 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d58b2536-eb2d-402e-ba58-47fc2a8e70bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:15.682317 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bba1ee4-6825-4055-ae1d-87da01aa636d admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:15.687933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083a5833-16a6-42c1-8de2-37892dbcdb90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:15.688729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083a5833-16a6-42c1-8de2-37892dbcdb90 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:15.689106 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083a5833-16a6-42c1-8de2-37892dbcdb90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:15.689195 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083a5833-16a6-42c1-8de2-37892dbcdb90 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:15.689838 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083a5833-16a6-42c1-8de2-37892dbcdb90 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:15.691512 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083a5833-16a6-42c1-8de2-37892dbcdb90 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:15.709609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1010/2004] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:15 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:15.710725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083a5833-16a6-42c1-8de2-37892dbcdb90 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:15.712589 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-083a5833-16a6-42c1-8de2-37892dbcdb90 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:15.726836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083a5833-16a6-42c1-8de2-37892dbcdb90 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:15.731798 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6703b852-700d-41ac-9e17-6a6cbf307e2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:15.732452 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6703b852-700d-41ac-9e17-6a6cbf307e2d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:15.732736 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6703b852-700d-41ac-9e17-6a6cbf307e2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:15.733014 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6703b852-700d-41ac-9e17-6a6cbf307e2d admin admin] PATH_INFO: `/v3/users/aac35b6e0efa4450ab5b15f60192439a` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:15.735131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6703b852-700d-41ac-9e17-6a6cbf307e2d admin admin] RBAC: Authorizing `identity:delete_user(user_id=aac35b6e0efa4450ab5b15f60192439a)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:15.738913 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58b2536-eb2d-402e-ba58-47fc2a8e70bf admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:15.741416 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6703b852-700d-41ac-9e17-6a6cbf307e2d admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:15.745743 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f788443-746c-44b2-823a-2139f598ca82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:15.746398 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f788443-746c-44b2-823a-2139f598ca82 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:15.746761 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f788443-746c-44b2-823a-2139f598ca82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:15.747037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f788443-746c-44b2-823a-2139f598ca82 admin admin] PATH_INFO: `/v3/projects/fed6c500a3bf409a8de067697b583a59` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:15.748218 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f788443-746c-44b2-823a-2139f598ca82 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fed6c500a3bf409a8de067697b583a59)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:15.750393 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f788443-746c-44b2-823a-2139f598ca82 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:15.763516 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:15.790395 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:INSERT=2,keystone:SELECT=90,keystone:DELETE=8 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:15.813628 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6f788443-746c-44b2-823a-2139f598ca82 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fed6c500a3bf409a8de067697b583a59', 'request_id': 'req-6f788443-746c-44b2-823a-2139f598ca82'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:15.818087 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-6703b852-700d-41ac-9e17-6a6cbf307e2d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aac35b6e0efa4450ab5b15f60192439a'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:15.819680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6f788443-746c-44b2-823a-2139f598ca82 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fed6c500a3bf409a8de067697b583a59', 'request_id': 'req-6f788443-746c-44b2-823a-2139f598ca82'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:15.827230 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-6703b852-700d-41ac-9e17-6a6cbf307e2d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aac35b6e0efa4450ab5b15f60192439a'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:15.832476 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6f788443-746c-44b2-823a-2139f598ca82 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fed6c500a3bf409a8de067697b583a59', 'request_id': 'req-6f788443-746c-44b2-823a-2139f598ca82'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:15.842158 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-6703b852-700d-41ac-9e17-6a6cbf307e2d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aac35b6e0efa4450ab5b15f60192439a'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:15.842974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-6703b852-700d-41ac-9e17-6a6cbf307e2d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aac35b6e0efa4450ab5b15f60192439a'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:15.847473 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6f788443-746c-44b2-823a-2139f598ca82 admin admin] The token cache is being invalidate because project fed6c500a3bf409a8de067697b583a59 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:15.847770 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6f788443-746c-44b2-823a-2139f598ca82 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:15.848459 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6f788443-746c-44b2-823a-2139f598ca82 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:15.850583 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 995/2005] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:15 2022] DELETE /identity/v3/projects/fed6c500a3bf409a8de067697b583a59 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:15.857070 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f788443-746c-44b2-823a-2139f598ca82 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:15.857070 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-6f788443-746c-44b2-823a-2139f598ca82 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:15.858588 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-6703b852-700d-41ac-9e17-6a6cbf307e2d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aac35b6e0efa4450ab5b15f60192439a'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:15.879772 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1011/2006] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:15 2022] DELETE /identity/v3/users/aac35b6e0efa4450ab5b15f60192439a => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:15.942668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f788443-746c-44b2-823a-2139f598ca82 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:16.052751 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee47126-e326-4ab2-8c8d-7084a1e58c1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:16.053652 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee47126-e326-4ab2-8c8d-7084a1e58c1f admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:16.054529 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee47126-e326-4ab2-8c8d-7084a1e58c1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:16.054815 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee47126-e326-4ab2-8c8d-7084a1e58c1f admin admin] PATH_INFO: `/v3/projects/525917f9910e403992fb40c1298a68a7/users/6d09a427b18a4be0b06a99eb9287e1a1/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:16.057296 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee47126-e326-4ab2-8c8d-7084a1e58c1f admin admin] RBAC: Authorizing `identity:create_grant(project_id=525917f9910e403992fb40c1298a68a7, user_id=6d09a427b18a4be0b06a99eb9287e1a1, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:16.064305 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee47126-e326-4ab2-8c8d-7084a1e58c1f admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:16.077244 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 996/2007] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:15 2022] PUT /identity/v3/projects/525917f9910e403992fb40c1298a68a7/users/6d09a427b18a4be0b06a99eb9287e1a1/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:16.088470 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca5045b-ad2f-4f44-bceb-68c9fd45b41b None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:16.088760 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca5045b-ad2f-4f44-bceb-68c9fd45b41b None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:16.089040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca5045b-ad2f-4f44-bceb-68c9fd45b41b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:16.120343 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-2ca5045b-ad2f-4f44-bceb-68c9fd45b41b None None] MFA Rules not processed for user `6d09a427b18a4be0b06a99eb9287e1a1`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:16.121557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-2ca5045b-ad2f-4f44-bceb-68c9fd45b41b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:16.210135 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1012/2008] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:16 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:16.218322 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee47126-e326-4ab2-8c8d-7084a1e58c1f admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:16.220649 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-dee47126-e326-4ab2-8c8d-7084a1e58c1f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:16.253238 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca5045b-ad2f-4f44-bceb-68c9fd45b41b None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:16.254718 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-2ca5045b-ad2f-4f44-bceb-68c9fd45b41b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:16.340276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee47126-e326-4ab2-8c8d-7084a1e58c1f admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:16.346415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4694a779-ae5f-466a-a1b3-cd7d57c96167 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:16.347308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4694a779-ae5f-466a-a1b3-cd7d57c96167 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:16.347727 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4694a779-ae5f-466a-a1b3-cd7d57c96167 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:16.348099 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4694a779-ae5f-466a-a1b3-cd7d57c96167 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:16.362519 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca5045b-ad2f-4f44-bceb-68c9fd45b41b None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:16.367005 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4694a779-ae5f-466a-a1b3-cd7d57c96167 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:16.368270 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e95f2b-2d0b-4179-8cea-5e5b83ccd811 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:16.368647 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4694a779-ae5f-466a-a1b3-cd7d57c96167 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:16.369066 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e95f2b-2d0b-4179-8cea-5e5b83ccd811 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:16.370259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e95f2b-2d0b-4179-8cea-5e5b83ccd811 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:16.371121 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e95f2b-2d0b-4179-8cea-5e5b83ccd811 admin admin] PATH_INFO: `/v3/projects/cfd8ac6e394049d79bfa3e24a525f293` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:16.372691 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e95f2b-2d0b-4179-8cea-5e5b83ccd811 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cfd8ac6e394049d79bfa3e24a525f293)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:16.374746 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e95f2b-2d0b-4179-8cea-5e5b83ccd811 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:16.375394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:DELETE=6,keystone:SELECT=92,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:16.400349 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 997/2009] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:16 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:16.418233 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:16.517110 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-05e95f2b-2d0b-4179-8cea-5e5b83ccd811 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cfd8ac6e394049d79bfa3e24a525f293', 'request_id': 'req-05e95f2b-2d0b-4179-8cea-5e5b83ccd811'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:16.531446 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-05e95f2b-2d0b-4179-8cea-5e5b83ccd811 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cfd8ac6e394049d79bfa3e24a525f293', 'request_id': 'req-05e95f2b-2d0b-4179-8cea-5e5b83ccd811'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:16.541661 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-05e95f2b-2d0b-4179-8cea-5e5b83ccd811 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cfd8ac6e394049d79bfa3e24a525f293', 'request_id': 'req-05e95f2b-2d0b-4179-8cea-5e5b83ccd811'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:16.556631 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-05e95f2b-2d0b-4179-8cea-5e5b83ccd811 admin admin] The token cache is being invalidate because project cfd8ac6e394049d79bfa3e24a525f293 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:16.556947 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-05e95f2b-2d0b-4179-8cea-5e5b83ccd811 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:16.557603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-05e95f2b-2d0b-4179-8cea-5e5b83ccd811 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:16.559336 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1013/2010] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:16 2022] DELETE /identity/v3/projects/cfd8ac6e394049d79bfa3e24a525f293 => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:16.565740 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c62e723-6f48-4e1d-83b1-a8865d563644 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:16.566031 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c62e723-6f48-4e1d-83b1-a8865d563644 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:16.566305 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c62e723-6f48-4e1d-83b1-a8865d563644 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:16.595745 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-5c62e723-6f48-4e1d-83b1-a8865d563644 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:16.597303 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-5c62e723-6f48-4e1d-83b1-a8865d563644 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:16.675238 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 998/2011] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:16.685489 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12d34e6-dc21-40ca-a8b3-d865e25a062c None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:16.685880 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12d34e6-dc21-40ca-a8b3-d865e25a062c None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:16.686255 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12d34e6-dc21-40ca-a8b3-d865e25a062c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:16.716401 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-e12d34e6-dc21-40ca-a8b3-d865e25a062c None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:16.717674 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-e12d34e6-dc21-40ca-a8b3-d865e25a062c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:16.728032 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1014/2012] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:16.732632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c62e723-6f48-4e1d-83b1-a8865d563644 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:16.746590 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c62e723-6f48-4e1d-83b1-a8865d563644 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:16.751595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432a0e7b-5e1a-41ca-ae8a-c58f46b02ba7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:16.752155 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432a0e7b-5e1a-41ca-ae8a-c58f46b02ba7 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:16.752428 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432a0e7b-5e1a-41ca-ae8a-c58f46b02ba7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:16.752693 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432a0e7b-5e1a-41ca-ae8a-c58f46b02ba7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:16.753443 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432a0e7b-5e1a-41ca-ae8a-c58f46b02ba7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:58:16.753779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432a0e7b-5e1a-41ca-ae8a-c58f46b02ba7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:16.755554 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432a0e7b-5e1a-41ca-ae8a-c58f46b02ba7 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:16.771621 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 999/2013] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:58:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:58:16.776655 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12d34e6-dc21-40ca-a8b3-d865e25a062c None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:16.787740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12d34e6-dc21-40ca-a8b3-d865e25a062c None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:16.793487 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46fb796-c8aa-4f63-a215-6eef1d60368d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:16.794334 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46fb796-c8aa-4f63-a215-6eef1d60368d admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:16.794821 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46fb796-c8aa-4f63-a215-6eef1d60368d admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:16.795214 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46fb796-c8aa-4f63-a215-6eef1d60368d admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:16.796084 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46fb796-c8aa-4f63-a215-6eef1d60368d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:16.799303 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46fb796-c8aa-4f63-a215-6eef1d60368d admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:16.834623 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1015/2014] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:16 2022] POST /identity/v3/projects => generated 377 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:16.839030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432a0e7b-5e1a-41ca-ae8a-c58f46b02ba7 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:16.856204 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432a0e7b-5e1a-41ca-ae8a-c58f46b02ba7 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:16.948061 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92af085-269b-44d1-adaa-237f8ecf9dfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:16.948061 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92af085-269b-44d1-adaa-237f8ecf9dfc admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:16.948061 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92af085-269b-44d1-adaa-237f8ecf9dfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:16.948061 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92af085-269b-44d1-adaa-237f8ecf9dfc admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:16.948402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92af085-269b-44d1-adaa-237f8ecf9dfc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:16.948402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92af085-269b-44d1-adaa-237f8ecf9dfc admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:16.987911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1000/2015] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:16 2022] POST /identity/v3/users => generated 289 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:16.988010 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46fb796-c8aa-4f63-a215-6eef1d60368d admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:17.005518 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46fb796-c8aa-4f63-a215-6eef1d60368d admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:17.011328 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c5686e-6c27-41aa-8635-5049935b902b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:17.011959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c5686e-6c27-41aa-8635-5049935b902b admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:17.012236 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c5686e-6c27-41aa-8635-5049935b902b admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:17.012514 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c5686e-6c27-41aa-8635-5049935b902b admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:17.013160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c5686e-6c27-41aa-8635-5049935b902b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:17.014956 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c5686e-6c27-41aa-8635-5049935b902b admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:17.033540 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1016/2016] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:16 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:58:17.034508 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92af085-269b-44d1-adaa-237f8ecf9dfc admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:17.046868 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92af085-269b-44d1-adaa-237f8ecf9dfc admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:17.054899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d2cf0e-bae0-4d25-aeb9-2fd4ce233b8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:17.055841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d2cf0e-bae0-4d25-aeb9-2fd4ce233b8f admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:17.056249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d2cf0e-bae0-4d25-aeb9-2fd4ce233b8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:17.056535 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d2cf0e-bae0-4d25-aeb9-2fd4ce233b8f admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:17.057232 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d2cf0e-bae0-4d25-aeb9-2fd4ce233b8f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:17.058934 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d2cf0e-bae0-4d25-aeb9-2fd4ce233b8f admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:17.073467 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1001/2017] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:17 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:58:17.077734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c5686e-6c27-41aa-8635-5049935b902b admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:17.093476 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c5686e-6c27-41aa-8635-5049935b902b admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:17.099728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27084dab-ae8a-44da-af9b-0f8402d383ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:17.102663 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27084dab-ae8a-44da-af9b-0f8402d383ad admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:17.103079 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27084dab-ae8a-44da-af9b-0f8402d383ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:17.103438 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27084dab-ae8a-44da-af9b-0f8402d383ad admin admin] PATH_INFO: `/v3/projects/716713d46b3743b49e9cbe9949a8c3de/users/6defda3dae6f4c1f919bdfecb8e66f26/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:17.129323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27084dab-ae8a-44da-af9b-0f8402d383ad admin admin] RBAC: Authorizing `identity:create_grant(project_id=716713d46b3743b49e9cbe9949a8c3de, user_id=6defda3dae6f4c1f919bdfecb8e66f26, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:17.135328 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27084dab-ae8a-44da-af9b-0f8402d383ad admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:17.183362 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1017/2018] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:17 2022] PUT /identity/v3/projects/716713d46b3743b49e9cbe9949a8c3de/users/6defda3dae6f4c1f919bdfecb8e66f26/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:17.195034 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502680e5-b070-44ed-822c-291acfb96810 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:17.195706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502680e5-b070-44ed-822c-291acfb96810 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:17.196217 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502680e5-b070-44ed-822c-291acfb96810 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:17.240935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-502680e5-b070-44ed-822c-291acfb96810 None None] MFA Rules not processed for user `6defda3dae6f4c1f919bdfecb8e66f26`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:17.242830 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-502680e5-b070-44ed-822c-291acfb96810 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:17.324591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:17.347896 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1002/2019] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:17 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:17.361812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27084dab-ae8a-44da-af9b-0f8402d383ad admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:17.363781 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-27084dab-ae8a-44da-af9b-0f8402d383ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:17.475814 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27084dab-ae8a-44da-af9b-0f8402d383ad admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:17.491518 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b6731e-1d40-4ee5-9359-bdf345d74f80 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7fe9d21099e4c4891c28687bb673214', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:17.492139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b6731e-1d40-4ee5-9359-bdf345d74f80 service glance] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:17.492562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b6731e-1d40-4ee5-9359-bdf345d74f80 service glance] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:17.492649 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b6731e-1d40-4ee5-9359-bdf345d74f80 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:17.511662 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b6731e-1d40-4ee5-9359-bdf345d74f80 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:17.520056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b6731e-1d40-4ee5-9359-bdf345d74f80 service glance] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:17.561447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1018/2020] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:17 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:18.501201 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502680e5-b070-44ed-822c-291acfb96810 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:18.504147 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-502680e5-b070-44ed-822c-291acfb96810 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:18.599414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502680e5-b070-44ed-822c-291acfb96810 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:18.605687 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c08906b-9055-4e6c-b1ee-94a3fedd43f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:18.606367 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c08906b-9055-4e6c-b1ee-94a3fedd43f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:18.606789 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c08906b-9055-4e6c-b1ee-94a3fedd43f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:18.607179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c08906b-9055-4e6c-b1ee-94a3fedd43f6 admin admin] PATH_INFO: `/v3/users/9c54c0c5f339483ab56d52bab2f63a17` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:18.608280 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c08906b-9055-4e6c-b1ee-94a3fedd43f6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9c54c0c5f339483ab56d52bab2f63a17)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:18.610219 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c08906b-9055-4e6c-b1ee-94a3fedd43f6 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:18.672518 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8c08906b-9055-4e6c-b1ee-94a3fedd43f6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9c54c0c5f339483ab56d52bab2f63a17'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:18.685493 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8c08906b-9055-4e6c-b1ee-94a3fedd43f6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9c54c0c5f339483ab56d52bab2f63a17'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:18.687747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8c08906b-9055-4e6c-b1ee-94a3fedd43f6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9c54c0c5f339483ab56d52bab2f63a17'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:18.688284 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8c08906b-9055-4e6c-b1ee-94a3fedd43f6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9c54c0c5f339483ab56d52bab2f63a17'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:18.699233 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8c08906b-9055-4e6c-b1ee-94a3fedd43f6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9c54c0c5f339483ab56d52bab2f63a17'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:18.718212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1003/2021] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:18 2022] DELETE /identity/v3/users/9c54c0c5f339483ab56d52bab2f63a17 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:18.722662 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848d7568-eab4-4e5e-a166-50b6548912ab None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:18.723137 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848d7568-eab4-4e5e-a166-50b6548912ab None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:18.723700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848d7568-eab4-4e5e-a166-50b6548912ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:18.745109 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-848d7568-eab4-4e5e-a166-50b6548912ab None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:18.746328 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-848d7568-eab4-4e5e-a166-50b6548912ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:18.793567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:DELETE=7,keystone:SELECT=90,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:18.814406 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1019/2022] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:18.821268 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c08906b-9055-4e6c-b1ee-94a3fedd43f6 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:18.828907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8c08906b-9055-4e6c-b1ee-94a3fedd43f6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:18.892177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c08906b-9055-4e6c-b1ee-94a3fedd43f6 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:18.896734 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2bcb01-12e3-4f6b-9740-2f6b4bd4780a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:18.897397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2bcb01-12e3-4f6b-9740-2f6b4bd4780a service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:18.897702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2bcb01-12e3-4f6b-9740-2f6b4bd4780a service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:18.897976 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2bcb01-12e3-4f6b-9740-2f6b4bd4780a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:18.907482 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2bcb01-12e3-4f6b-9740-2f6b4bd4780a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:18.908919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2bcb01-12e3-4f6b-9740-2f6b4bd4780a service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:18.934544 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1004/2023] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:58:19.420315 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848d7568-eab4-4e5e-a166-50b6548912ab None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:19.422097 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-848d7568-eab4-4e5e-a166-50b6548912ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:19.477660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848d7568-eab4-4e5e-a166-50b6548912ab None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:19.497401 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baeeb5e2-4efb-444d-8c0a-8bf59410f755 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:19.498070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baeeb5e2-4efb-444d-8c0a-8bf59410f755 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:19.498339 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baeeb5e2-4efb-444d-8c0a-8bf59410f755 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:19.498961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baeeb5e2-4efb-444d-8c0a-8bf59410f755 admin admin] PATH_INFO: `/v3/projects/c6d771c1e2fe4b1b99982301423f0f13` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:19.500918 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baeeb5e2-4efb-444d-8c0a-8bf59410f755 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c6d771c1e2fe4b1b99982301423f0f13)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:19.503056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baeeb5e2-4efb-444d-8c0a-8bf59410f755 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:19.609492 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-baeeb5e2-4efb-444d-8c0a-8bf59410f755 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c6d771c1e2fe4b1b99982301423f0f13', 'request_id': 'req-baeeb5e2-4efb-444d-8c0a-8bf59410f755'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:19.617006 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-baeeb5e2-4efb-444d-8c0a-8bf59410f755 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c6d771c1e2fe4b1b99982301423f0f13', 'request_id': 'req-baeeb5e2-4efb-444d-8c0a-8bf59410f755'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:19.634077 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-baeeb5e2-4efb-444d-8c0a-8bf59410f755 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c6d771c1e2fe4b1b99982301423f0f13', 'request_id': 'req-baeeb5e2-4efb-444d-8c0a-8bf59410f755'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:19.653170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-baeeb5e2-4efb-444d-8c0a-8bf59410f755 admin admin] The token cache is being invalidate because project c6d771c1e2fe4b1b99982301423f0f13 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:19.653460 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-baeeb5e2-4efb-444d-8c0a-8bf59410f755 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:19.654796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-baeeb5e2-4efb-444d-8c0a-8bf59410f755 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:19.656468 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1020/2024] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:19 2022] DELETE /identity/v3/projects/c6d771c1e2fe4b1b99982301423f0f13 => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:19.660621 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2bcb01-12e3-4f6b-9740-2f6b4bd4780a service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:19.679200 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2bcb01-12e3-4f6b-9740-2f6b4bd4780a service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:19.697311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:19.748077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242f3135-e9c2-4061-88a0-39e92a27e30a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:19.748728 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242f3135-e9c2-4061-88a0-39e92a27e30a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:19.749000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242f3135-e9c2-4061-88a0-39e92a27e30a admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:19.749271 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242f3135-e9c2-4061-88a0-39e92a27e30a admin admin] PATH_INFO: `/v3/users/6d09a427b18a4be0b06a99eb9287e1a1` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:19.750493 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242f3135-e9c2-4061-88a0-39e92a27e30a admin admin] RBAC: Authorizing `identity:delete_user(user_id=6d09a427b18a4be0b06a99eb9287e1a1)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:19.752505 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242f3135-e9c2-4061-88a0-39e92a27e30a admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:19.806166 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-242f3135-e9c2-4061-88a0-39e92a27e30a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6d09a427b18a4be0b06a99eb9287e1a1'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:19.822178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-242f3135-e9c2-4061-88a0-39e92a27e30a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6d09a427b18a4be0b06a99eb9287e1a1'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:19.823573 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-242f3135-e9c2-4061-88a0-39e92a27e30a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6d09a427b18a4be0b06a99eb9287e1a1'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:19.824182 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-242f3135-e9c2-4061-88a0-39e92a27e30a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6d09a427b18a4be0b06a99eb9287e1a1'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:19.832156 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-242f3135-e9c2-4061-88a0-39e92a27e30a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6d09a427b18a4be0b06a99eb9287e1a1'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:19.851467 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1005/2025] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:19 2022] DELETE /identity/v3/users/6d09a427b18a4be0b06a99eb9287e1a1 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:20.193492 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baeeb5e2-4efb-444d-8c0a-8bf59410f755 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:20.195700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-baeeb5e2-4efb-444d-8c0a-8bf59410f755 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:20.220758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baeeb5e2-4efb-444d-8c0a-8bf59410f755 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:20.319363 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc5f73e-a99f-4b26-a40e-267de83b206e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:20.320334 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc5f73e-a99f-4b26-a40e-267de83b206e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:20.320612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc5f73e-a99f-4b26-a40e-267de83b206e admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:20.320896 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc5f73e-a99f-4b26-a40e-267de83b206e admin admin] PATH_INFO: `/v3/projects/525917f9910e403992fb40c1298a68a7` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:20.322106 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc5f73e-a99f-4b26-a40e-267de83b206e admin admin] RBAC: Authorizing `identity:delete_project(project_id=525917f9910e403992fb40c1298a68a7)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:20.324364 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc5f73e-a99f-4b26-a40e-267de83b206e admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:20.412956 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-bbc5f73e-a99f-4b26-a40e-267de83b206e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '525917f9910e403992fb40c1298a68a7', 'request_id': 'req-bbc5f73e-a99f-4b26-a40e-267de83b206e'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:20.421224 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-bbc5f73e-a99f-4b26-a40e-267de83b206e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '525917f9910e403992fb40c1298a68a7', 'request_id': 'req-bbc5f73e-a99f-4b26-a40e-267de83b206e'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:20.436555 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-bbc5f73e-a99f-4b26-a40e-267de83b206e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '525917f9910e403992fb40c1298a68a7', 'request_id': 'req-bbc5f73e-a99f-4b26-a40e-267de83b206e'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:20.454841 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-bbc5f73e-a99f-4b26-a40e-267de83b206e admin admin] The token cache is being invalidate because project 525917f9910e403992fb40c1298a68a7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:20.455129 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-bbc5f73e-a99f-4b26-a40e-267de83b206e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:20.456404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-bbc5f73e-a99f-4b26-a40e-267de83b206e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:20.457982 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1021/2026] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:20 2022] DELETE /identity/v3/projects/525917f9910e403992fb40c1298a68a7 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:20.463522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d9623f-33f8-4b49-95bc-086a8d21949f None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:20.463814 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d9623f-33f8-4b49-95bc-086a8d21949f None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:20.464092 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d9623f-33f8-4b49-95bc-086a8d21949f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:20.492561 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-92d9623f-33f8-4b49-95bc-086a8d21949f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:20.493925 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-92d9623f-33f8-4b49-95bc-086a8d21949f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:20.620889 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1006/2027] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:20.628409 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c079b73f-3c18-4a3f-b9ec-68c1498c5c5d None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:20.628745 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c079b73f-3c18-4a3f-b9ec-68c1498c5c5d None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:20.629027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c079b73f-3c18-4a3f-b9ec-68c1498c5c5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:20.639045 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:20.657940 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-c079b73f-3c18-4a3f-b9ec-68c1498c5c5d None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:20.663281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c079b73f-3c18-4a3f-b9ec-68c1498c5c5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:20.683242 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1022/2028] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:20.689365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d9623f-33f8-4b49-95bc-086a8d21949f None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:20.702683 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d9623f-33f8-4b49-95bc-086a8d21949f None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:20.712452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ce3bb3-3f41-4fb5-a1bf-6724d34e0b83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:20.714036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ce3bb3-3f41-4fb5-a1bf-6724d34e0b83 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:20.714036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ce3bb3-3f41-4fb5-a1bf-6724d34e0b83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:20.714036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ce3bb3-3f41-4fb5-a1bf-6724d34e0b83 admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:20.715022 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ce3bb3-3f41-4fb5-a1bf-6724d34e0b83 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:58:20.715390 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ce3bb3-3f41-4fb5-a1bf-6724d34e0b83 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:20.717363 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ce3bb3-3f41-4fb5-a1bf-6724d34e0b83 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:20.739116 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1007/2029] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:58:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:58:20.743906 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c079b73f-3c18-4a3f-b9ec-68c1498c5c5d None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:20.753467 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c079b73f-3c18-4a3f-b9ec-68c1498c5c5d None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:20.758432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded6225b-1c38-4cb7-ac36-49edfaef236b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:20.759169 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded6225b-1c38-4cb7-ac36-49edfaef236b admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:20.759503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded6225b-1c38-4cb7-ac36-49edfaef236b admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:20.759840 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded6225b-1c38-4cb7-ac36-49edfaef236b admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:20.760548 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded6225b-1c38-4cb7-ac36-49edfaef236b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:20.762572 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded6225b-1c38-4cb7-ac36-49edfaef236b admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:20.799849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1023/2030] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:20 2022] POST /identity/v3/projects => generated 369 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:20.800990 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ce3bb3-3f41-4fb5-a1bf-6724d34e0b83 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:20.811815 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ce3bb3-3f41-4fb5-a1bf-6724d34e0b83 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:20.830796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:20.872314 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922eab42-eda7-4795-90a8-9fc3c079c5c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:20.872925 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922eab42-eda7-4795-90a8-9fc3c079c5c0 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:20.873208 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922eab42-eda7-4795-90a8-9fc3c079c5c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:20.873485 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922eab42-eda7-4795-90a8-9fc3c079c5c0 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:20.874120 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922eab42-eda7-4795-90a8-9fc3c079c5c0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:20.876065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922eab42-eda7-4795-90a8-9fc3c079c5c0 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:20.896500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1008/2031] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:20 2022] POST /identity/v3/users => generated 285 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:58:20.900783 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded6225b-1c38-4cb7-ac36-49edfaef236b admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:20.911097 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded6225b-1c38-4cb7-ac36-49edfaef236b admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:20.916395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6347dd84-aa59-478a-88a9-9c92c43d2047 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:20.916996 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6347dd84-aa59-478a-88a9-9c92c43d2047 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:20.917274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6347dd84-aa59-478a-88a9-9c92c43d2047 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:20.917547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6347dd84-aa59-478a-88a9-9c92c43d2047 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:20.918186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6347dd84-aa59-478a-88a9-9c92c43d2047 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:20.919801 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6347dd84-aa59-478a-88a9-9c92c43d2047 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:20.930210 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1024/2032] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:20.934367 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922eab42-eda7-4795-90a8-9fc3c079c5c0 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:20.944635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922eab42-eda7-4795-90a8-9fc3c079c5c0 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:20.949576 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e0d8af-492a-4bc4-b03f-e67a2f75926b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:20.950183 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e0d8af-492a-4bc4-b03f-e67a2f75926b admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:20.950486 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e0d8af-492a-4bc4-b03f-e67a2f75926b admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:20.950759 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e0d8af-492a-4bc4-b03f-e67a2f75926b admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:20.951416 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e0d8af-492a-4bc4-b03f-e67a2f75926b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:20.953108 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e0d8af-492a-4bc4-b03f-e67a2f75926b admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:20.965956 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1009/2033] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:20 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:20.970111 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6347dd84-aa59-478a-88a9-9c92c43d2047 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:20.980831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6347dd84-aa59-478a-88a9-9c92c43d2047 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:20.985424 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e6d942-dca0-41a2-8233-ebc387c4c0d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:20.986072 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e6d942-dca0-41a2-8233-ebc387c4c0d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:20.986353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e6d942-dca0-41a2-8233-ebc387c4c0d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:20.986663 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e6d942-dca0-41a2-8233-ebc387c4c0d9 admin admin] PATH_INFO: `/v3/projects/5ebb3645140241a6bc1ca99cf4c8f6ad/users/2e5b43a704df4fbd9cfbc1028d787b98/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:21.001706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e6d942-dca0-41a2-8233-ebc387c4c0d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5ebb3645140241a6bc1ca99cf4c8f6ad, user_id=2e5b43a704df4fbd9cfbc1028d787b98, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:21.004483 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e6d942-dca0-41a2-8233-ebc387c4c0d9 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:21.019584 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1025/2034] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:20 2022] PUT /identity/v3/projects/5ebb3645140241a6bc1ca99cf4c8f6ad/users/2e5b43a704df4fbd9cfbc1028d787b98/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:21.030498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b912467-0fb5-45d0-b9b9-6efd309d53c0 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:21.030857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b912467-0fb5-45d0-b9b9-6efd309d53c0 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:21.031236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b912467-0fb5-45d0-b9b9-6efd309d53c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:21.061149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-8b912467-0fb5-45d0-b9b9-6efd309d53c0 None None] MFA Rules not processed for user `2e5b43a704df4fbd9cfbc1028d787b98`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:21.062374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8b912467-0fb5-45d0-b9b9-6efd309d53c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:21.134355 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1010/2035] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:21 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:21.139800 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e6d942-dca0-41a2-8233-ebc387c4c0d9 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:21.202771 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e6d942-dca0-41a2-8233-ebc387c4c0d9 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:21.207473 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c73920a-05be-4799-bd0f-111d07533dcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:21.208090 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c73920a-05be-4799-bd0f-111d07533dcc admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:21.208957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c73920a-05be-4799-bd0f-111d07533dcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:21.209265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c73920a-05be-4799-bd0f-111d07533dcc admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:21.209908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c73920a-05be-4799-bd0f-111d07533dcc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:21.211864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c73920a-05be-4799-bd0f-111d07533dcc admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:21.241140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1026/2036] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:21 2022] POST /identity/v3/projects => generated 369 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:21.245117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b912467-0fb5-45d0-b9b9-6efd309d53c0 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:21.256655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b912467-0fb5-45d0-b9b9-6efd309d53c0 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:21.314861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a00a63-ce14-41e3-946d-7663662237d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:21.315466 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a00a63-ce14-41e3-946d-7663662237d6 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:21.315744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a00a63-ce14-41e3-946d-7663662237d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:21.316011 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a00a63-ce14-41e3-946d-7663662237d6 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:21.316634 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a00a63-ce14-41e3-946d-7663662237d6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:21.319075 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a00a63-ce14-41e3-946d-7663662237d6 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:21.341385 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1011/2037] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:21 2022] POST /identity/v3/users => generated 291 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:21.345598 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c73920a-05be-4799-bd0f-111d07533dcc admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:21.357579 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c73920a-05be-4799-bd0f-111d07533dcc admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:21.362864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b66764a-bcf1-4d27-aa95-2f808b0e73ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:21.363485 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b66764a-bcf1-4d27-aa95-2f808b0e73ec admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:21.363762 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b66764a-bcf1-4d27-aa95-2f808b0e73ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:21.364036 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b66764a-bcf1-4d27-aa95-2f808b0e73ec admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:21.364718 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b66764a-bcf1-4d27-aa95-2f808b0e73ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:21.366336 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b66764a-bcf1-4d27-aa95-2f808b0e73ec admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:21.377384 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1027/2038] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:21 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:21.382108 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a00a63-ce14-41e3-946d-7663662237d6 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:21.392148 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a00a63-ce14-41e3-946d-7663662237d6 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:21.397151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87060415-ba19-405f-8927-804e1fedb6c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:21.397807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87060415-ba19-405f-8927-804e1fedb6c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:21.398073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87060415-ba19-405f-8927-804e1fedb6c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:21.398343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87060415-ba19-405f-8927-804e1fedb6c6 admin admin] PATH_INFO: `/v3/projects/ba17f3156ae24556b81a64012a8d4cbc/users/a2c0b2a9bb104ef78ba29344336c7027/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:21.414154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87060415-ba19-405f-8927-804e1fedb6c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ba17f3156ae24556b81a64012a8d4cbc, user_id=a2c0b2a9bb104ef78ba29344336c7027, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:21.416828 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87060415-ba19-405f-8927-804e1fedb6c6 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:21.420814 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:21.435559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1012/2039] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:21 2022] PUT /identity/v3/projects/ba17f3156ae24556b81a64012a8d4cbc/users/a2c0b2a9bb104ef78ba29344336c7027/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:21.438632 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b66764a-bcf1-4d27-aa95-2f808b0e73ec admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:21.449611 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b66764a-bcf1-4d27-aa95-2f808b0e73ec admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:21.506624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543a3fce-1f11-4a8e-ba75-579493539b51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:21.507125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543a3fce-1f11-4a8e-ba75-579493539b51 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:21.507400 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543a3fce-1f11-4a8e-ba75-579493539b51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:21.507670 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543a3fce-1f11-4a8e-ba75-579493539b51 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:21.508308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543a3fce-1f11-4a8e-ba75-579493539b51 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:21.510290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543a3fce-1f11-4a8e-ba75-579493539b51 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:21.523927 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1028/2040] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:21 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:21.525488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87060415-ba19-405f-8927-804e1fedb6c6 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:21.542718 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87060415-ba19-405f-8927-804e1fedb6c6 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:21.548780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a95fbb-a2d1-4cfb-84e6-438f87edc36b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:21.549444 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a95fbb-a2d1-4cfb-84e6-438f87edc36b admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:21.549722 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a95fbb-a2d1-4cfb-84e6-438f87edc36b admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:21.549996 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a95fbb-a2d1-4cfb-84e6-438f87edc36b admin admin] PATH_INFO: `/v3/projects/ba17f3156ae24556b81a64012a8d4cbc/users/a2c0b2a9bb104ef78ba29344336c7027/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:21.552707 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a95fbb-a2d1-4cfb-84e6-438f87edc36b admin admin] RBAC: Authorizing `identity:create_grant(project_id=ba17f3156ae24556b81a64012a8d4cbc, user_id=a2c0b2a9bb104ef78ba29344336c7027, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:21.555460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a95fbb-a2d1-4cfb-84e6-438f87edc36b admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:21.568247 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1013/2041] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:21 2022] PUT /identity/v3/projects/ba17f3156ae24556b81a64012a8d4cbc/users/a2c0b2a9bb104ef78ba29344336c7027/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:21.579101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015496e6-5177-4cb8-a97e-f06ef1959e30 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:21.579483 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015496e6-5177-4cb8-a97e-f06ef1959e30 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:21.579685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015496e6-5177-4cb8-a97e-f06ef1959e30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:21.601012 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:21.613513 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-015496e6-5177-4cb8-a97e-f06ef1959e30 None None] MFA Rules not processed for user `a2c0b2a9bb104ef78ba29344336c7027`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:21.615746 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-015496e6-5177-4cb8-a97e-f06ef1959e30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:21.689338 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1029/2042] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:21 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:21.700855 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a95fbb-a2d1-4cfb-84e6-438f87edc36b admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:21.702737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-e2a95fbb-a2d1-4cfb-84e6-438f87edc36b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:21.762114 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a95fbb-a2d1-4cfb-84e6-438f87edc36b admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:21.765717 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca46f49-0f96-469d-bbdc-9225a39627b5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:21.766338 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca46f49-0f96-469d-bbdc-9225a39627b5 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:21.766644 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca46f49-0f96-469d-bbdc-9225a39627b5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:21.766916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca46f49-0f96-469d-bbdc-9225a39627b5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:21.776634 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca46f49-0f96-469d-bbdc-9225a39627b5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:21.778059 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca46f49-0f96-469d-bbdc-9225a39627b5 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:21.789183 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1014/2043] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:21 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:21.855614 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015496e6-5177-4cb8-a97e-f06ef1959e30 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:21.857846 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-015496e6-5177-4cb8-a97e-f06ef1959e30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:21.880300 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015496e6-5177-4cb8-a97e-f06ef1959e30 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:21.884117 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba06c7b1-5781-4bdc-88d3-205d90691d52 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:21.884861 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba06c7b1-5781-4bdc-88d3-205d90691d52 service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:21.885211 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba06c7b1-5781-4bdc-88d3-205d90691d52 service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:21.885622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba06c7b1-5781-4bdc-88d3-205d90691d52 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:21.950108 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba06c7b1-5781-4bdc-88d3-205d90691d52 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:21.951943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba06c7b1-5781-4bdc-88d3-205d90691d52 service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:21.966449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1030/2044] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:21 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:25.661423 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0b8828-55f7-44e9-a2ab-5f0f264016d8 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:25.662231 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0b8828-55f7-44e9-a2ab-5f0f264016d8 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:25.662574 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0b8828-55f7-44e9-a2ab-5f0f264016d8 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:25.663963 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1015/2045] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 18 00:58:25 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 00:58:25.683689 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca73ef79-1be4-4708-8c23-d24efb0724d9 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:25.684480 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca73ef79-1be4-4708-8c23-d24efb0724d9 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:25.684764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca73ef79-1be4-4708-8c23-d24efb0724d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:25.686180 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0b8828-55f7-44e9-a2ab-5f0f264016d8 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:25.736202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-ca73ef79-1be4-4708-8c23-d24efb0724d9 None None] MFA Rules not processed for user `8f30116fb5ce43dab69efa8bafd9d049`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:25.738041 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-ca73ef79-1be4-4708-8c23-d24efb0724d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:25.787031 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0b8828-55f7-44e9-a2ab-5f0f264016d8 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:25.791722 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467b52d1-82e5-4d6c-a3c6-e49bb13f4c9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:25.792361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467b52d1-82e5-4d6c-a3c6-e49bb13f4c9b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:25.792641 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467b52d1-82e5-4d6c-a3c6-e49bb13f4c9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:25.792916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467b52d1-82e5-4d6c-a3c6-e49bb13f4c9b admin admin] PATH_INFO: `/v3/users/2e5b43a704df4fbd9cfbc1028d787b98` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:25.794047 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467b52d1-82e5-4d6c-a3c6-e49bb13f4c9b admin admin] RBAC: Authorizing `identity:delete_user(user_id=2e5b43a704df4fbd9cfbc1028d787b98)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:25.796005 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467b52d1-82e5-4d6c-a3c6-e49bb13f4c9b admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:25.818280 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1031/2046] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 18 00:58:25 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:25.819617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca73ef79-1be4-4708-8c23-d24efb0724d9 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:25.821065 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-ca73ef79-1be4-4708-8c23-d24efb0724d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:25.836647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca73ef79-1be4-4708-8c23-d24efb0724d9 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:25.841294 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5018a67-55d8-44e5-a042-04be1763d325 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:25.842259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5018a67-55d8-44e5-a042-04be1763d325 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:25.842371 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5018a67-55d8-44e5-a042-04be1763d325 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:25.842657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5018a67-55d8-44e5-a042-04be1763d325 admin admin] PATH_INFO: `/v3/users/468d75d910874fa8bb0f4dacb589f64c` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:25.843814 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5018a67-55d8-44e5-a042-04be1763d325 admin admin] RBAC: Authorizing `identity:delete_user(user_id=468d75d910874fa8bb0f4dacb589f64c)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:25.845745 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5018a67-55d8-44e5-a042-04be1763d325 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:25.859785 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-467b52d1-82e5-4d6c-a3c6-e49bb13f4c9b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2e5b43a704df4fbd9cfbc1028d787b98'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:25.873908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:25.878918 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-467b52d1-82e5-4d6c-a3c6-e49bb13f4c9b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2e5b43a704df4fbd9cfbc1028d787b98'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:25.879761 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-467b52d1-82e5-4d6c-a3c6-e49bb13f4c9b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2e5b43a704df4fbd9cfbc1028d787b98'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:25.880583 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-467b52d1-82e5-4d6c-a3c6-e49bb13f4c9b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2e5b43a704df4fbd9cfbc1028d787b98'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:25.889866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-467b52d1-82e5-4d6c-a3c6-e49bb13f4c9b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2e5b43a704df4fbd9cfbc1028d787b98'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:25.903630 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-c5018a67-55d8-44e5-a042-04be1763d325 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '468d75d910874fa8bb0f4dacb589f64c'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:25.913356 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-c5018a67-55d8-44e5-a042-04be1763d325 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '468d75d910874fa8bb0f4dacb589f64c'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:25.918306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1016/2047] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:25 2022] DELETE /identity/v3/users/2e5b43a704df4fbd9cfbc1028d787b98 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:25.922002 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467b52d1-82e5-4d6c-a3c6-e49bb13f4c9b admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:25.923450 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-467b52d1-82e5-4d6c-a3c6-e49bb13f4c9b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:25.926278 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-c5018a67-55d8-44e5-a042-04be1763d325 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '468d75d910874fa8bb0f4dacb589f64c'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:25.927560 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-c5018a67-55d8-44e5-a042-04be1763d325 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '468d75d910874fa8bb0f4dacb589f64c'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:25.931039 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:25.936956 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-c5018a67-55d8-44e5-a042-04be1763d325 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '468d75d910874fa8bb0f4dacb589f64c'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:25.944527 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1032/2048] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:25 2022] DELETE /identity/v3/users/468d75d910874fa8bb0f4dacb589f64c => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:25.945761 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740418cd-cd76-43be-81e9-a08fb8f3e26c None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:25.946039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740418cd-cd76-43be-81e9-a08fb8f3e26c None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:25.946318 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740418cd-cd76-43be-81e9-a08fb8f3e26c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:25.966844 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-740418cd-cd76-43be-81e9-a08fb8f3e26c None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:25.968061 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-740418cd-cd76-43be-81e9-a08fb8f3e26c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:26.009375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467b52d1-82e5-4d6c-a3c6-e49bb13f4c9b admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:26.045809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1033/2049] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:26.057407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740418cd-cd76-43be-81e9-a08fb8f3e26c None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:26.058935 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-740418cd-cd76-43be-81e9-a08fb8f3e26c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:26.085365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f196c602-15ff-4937-957e-8e4a7380eaa2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8f30116fb5ce43dab69efa8bafd9d049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:26.085365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f196c602-15ff-4937-957e-8e4a7380eaa2 service placement] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:26.085640 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f196c602-15ff-4937-957e-8e4a7380eaa2 service placement] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:26.086262 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f196c602-15ff-4937-957e-8e4a7380eaa2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:26.088216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-f196c602-15ff-4937-957e-8e4a7380eaa2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:26.106045 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f196c602-15ff-4937-957e-8e4a7380eaa2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:26.107727 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f196c602-15ff-4937-957e-8e4a7380eaa2 service placement] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:26.119213 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1017/2050] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:25 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:26.134983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740418cd-cd76-43be-81e9-a08fb8f3e26c None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:26.138898 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90842d83-6a51-4c1d-ac18-9dc8671d29c2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:26.139664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90842d83-6a51-4c1d-ac18-9dc8671d29c2 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:26.139953 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90842d83-6a51-4c1d-ac18-9dc8671d29c2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:26.140249 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90842d83-6a51-4c1d-ac18-9dc8671d29c2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:26.146812 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f196c602-15ff-4937-957e-8e4a7380eaa2 service placement] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:26.148968 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-f196c602-15ff-4937-957e-8e4a7380eaa2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:26.153555 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90842d83-6a51-4c1d-ac18-9dc8671d29c2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:26.155132 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90842d83-6a51-4c1d-ac18-9dc8671d29c2 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:26.167104 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1034/2051] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:58:26.168974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f196c602-15ff-4937-957e-8e4a7380eaa2 service placement] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:26.173952 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d53bee6-9e2f-4890-91c0-7329a969df74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:26.175113 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d53bee6-9e2f-4890-91c0-7329a969df74 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:26.175407 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d53bee6-9e2f-4890-91c0-7329a969df74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:26.175683 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d53bee6-9e2f-4890-91c0-7329a969df74 admin admin] PATH_INFO: `/v3/projects/b94600e628984a14a6abc8764ea5be31` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:26.177490 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d53bee6-9e2f-4890-91c0-7329a969df74 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b94600e628984a14a6abc8764ea5be31)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:26.179449 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d53bee6-9e2f-4890-91c0-7329a969df74 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:26.241530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-1d53bee6-9e2f-4890-91c0-7329a969df74 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b94600e628984a14a6abc8764ea5be31', 'request_id': 'req-1d53bee6-9e2f-4890-91c0-7329a969df74'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:26.248490 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-1d53bee6-9e2f-4890-91c0-7329a969df74 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b94600e628984a14a6abc8764ea5be31', 'request_id': 'req-1d53bee6-9e2f-4890-91c0-7329a969df74'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:26.256222 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-1d53bee6-9e2f-4890-91c0-7329a969df74 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b94600e628984a14a6abc8764ea5be31', 'request_id': 'req-1d53bee6-9e2f-4890-91c0-7329a969df74'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:26.268212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-1d53bee6-9e2f-4890-91c0-7329a969df74 admin admin] The token cache is being invalidate because project b94600e628984a14a6abc8764ea5be31 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:26.268502 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-1d53bee6-9e2f-4890-91c0-7329a969df74 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:26.269301 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-1d53bee6-9e2f-4890-91c0-7329a969df74 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:26.271388 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1018/2052] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:26 2022] DELETE /identity/v3/projects/b94600e628984a14a6abc8764ea5be31 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:26.276510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e816ba9d-403a-408e-b027-49fb4cfbc662 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:26.276803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e816ba9d-403a-408e-b027-49fb4cfbc662 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:26.277115 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e816ba9d-403a-408e-b027-49fb4cfbc662 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:26.298046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-e816ba9d-403a-408e-b027-49fb4cfbc662 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:26.299341 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-e816ba9d-403a-408e-b027-49fb4cfbc662 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:26.349464 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:26.375772 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1035/2053] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:26.383907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d019bd-733d-4170-ba45-0f6124fe6bb8 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:26.384301 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d019bd-733d-4170-ba45-0f6124fe6bb8 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:26.384682 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d019bd-733d-4170-ba45-0f6124fe6bb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:26.405717 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-61d019bd-733d-4170-ba45-0f6124fe6bb8 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:26.407210 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-61d019bd-733d-4170-ba45-0f6124fe6bb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:26.418500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1019/2054] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:26.420721 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e816ba9d-403a-408e-b027-49fb4cfbc662 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:26.431300 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e816ba9d-403a-408e-b027-49fb4cfbc662 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:26.435990 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cf4a02-ab2a-47ff-b365-b308ce943e87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:26.436998 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cf4a02-ab2a-47ff-b365-b308ce943e87 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:26.437112 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cf4a02-ab2a-47ff-b365-b308ce943e87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:26.437474 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cf4a02-ab2a-47ff-b365-b308ce943e87 admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:26.438224 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17cf4a02-ab2a-47ff-b365-b308ce943e87 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:58:26.439876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17cf4a02-ab2a-47ff-b365-b308ce943e87 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:26.441673 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17cf4a02-ab2a-47ff-b365-b308ce943e87 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:26.455951 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1036/2055] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:58:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:58:26.461778 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d019bd-733d-4170-ba45-0f6124fe6bb8 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:26.470355 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cf4a02-ab2a-47ff-b365-b308ce943e87 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:26.471771 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d019bd-733d-4170-ba45-0f6124fe6bb8 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:26.474552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-17cf4a02-ab2a-47ff-b365-b308ce943e87 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:26.477927 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d01085-fc90-4de9-9205-3292c5d91a51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:26.478264 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d01085-fc90-4de9-9205-3292c5d91a51 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:26.478582 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d01085-fc90-4de9-9205-3292c5d91a51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:26.478854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d01085-fc90-4de9-9205-3292c5d91a51 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:26.479515 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d01085-fc90-4de9-9205-3292c5d91a51 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:26.481454 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d01085-fc90-4de9-9205-3292c5d91a51 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:26.488682 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cf4a02-ab2a-47ff-b365-b308ce943e87 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:26.494670 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79917758-a576-4fcc-9fd5-1a7d544dd21a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:26.495411 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79917758-a576-4fcc-9fd5-1a7d544dd21a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:26.495756 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79917758-a576-4fcc-9fd5-1a7d544dd21a admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:26.496040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79917758-a576-4fcc-9fd5-1a7d544dd21a admin admin] PATH_INFO: `/v3/projects/5ebb3645140241a6bc1ca99cf4c8f6ad` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:26.497185 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79917758-a576-4fcc-9fd5-1a7d544dd21a admin admin] RBAC: Authorizing `identity:delete_project(project_id=5ebb3645140241a6bc1ca99cf4c8f6ad)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:26.499182 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79917758-a576-4fcc-9fd5-1a7d544dd21a admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:26.507887 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:26.543140 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1020/2056] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:26 2022] POST /identity/v3/projects => generated 371 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:58:26.544469 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d01085-fc90-4de9-9205-3292c5d91a51 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:26.575132 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-79917758-a576-4fcc-9fd5-1a7d544dd21a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5ebb3645140241a6bc1ca99cf4c8f6ad', 'request_id': 'req-79917758-a576-4fcc-9fd5-1a7d544dd21a'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:26.581684 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-79917758-a576-4fcc-9fd5-1a7d544dd21a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5ebb3645140241a6bc1ca99cf4c8f6ad', 'request_id': 'req-79917758-a576-4fcc-9fd5-1a7d544dd21a'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:26.590680 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-79917758-a576-4fcc-9fd5-1a7d544dd21a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5ebb3645140241a6bc1ca99cf4c8f6ad', 'request_id': 'req-79917758-a576-4fcc-9fd5-1a7d544dd21a'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:26.604849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-79917758-a576-4fcc-9fd5-1a7d544dd21a admin admin] The token cache is being invalidate because project 5ebb3645140241a6bc1ca99cf4c8f6ad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:26.605138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-79917758-a576-4fcc-9fd5-1a7d544dd21a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:26.605937 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-79917758-a576-4fcc-9fd5-1a7d544dd21a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:26.607359 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1037/2057] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:26 2022] DELETE /identity/v3/projects/5ebb3645140241a6bc1ca99cf4c8f6ad => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:26.611241 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79917758-a576-4fcc-9fd5-1a7d544dd21a admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:26.612632 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-79917758-a576-4fcc-9fd5-1a7d544dd21a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:26.625400 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d01085-fc90-4de9-9205-3292c5d91a51 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:26.686167 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79917758-a576-4fcc-9fd5-1a7d544dd21a admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:26.691953 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97b2371-0674-431c-9fdd-444cea05fb2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:26.693127 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97b2371-0674-431c-9fdd-444cea05fb2e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:26.694722 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97b2371-0674-431c-9fdd-444cea05fb2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:26.695019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97b2371-0674-431c-9fdd-444cea05fb2e admin admin] PATH_INFO: `/v3/users/a2c0b2a9bb104ef78ba29344336c7027` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:26.696104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2916a9b2-52c4-4a67-a41b-78691c711581 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:26.696712 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97b2371-0674-431c-9fdd-444cea05fb2e admin admin] RBAC: Authorizing `identity:delete_user(user_id=a2c0b2a9bb104ef78ba29344336c7027)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:26.696821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2916a9b2-52c4-4a67-a41b-78691c711581 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:26.697115 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2916a9b2-52c4-4a67-a41b-78691c711581 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:26.697393 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2916a9b2-52c4-4a67-a41b-78691c711581 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:26.698335 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2916a9b2-52c4-4a67-a41b-78691c711581 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:26.698943 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97b2371-0674-431c-9fdd-444cea05fb2e admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:26.701296 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2916a9b2-52c4-4a67-a41b-78691c711581 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:26.741234 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1021/2058] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:26 2022] POST /identity/v3/users => generated 286 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:58:26.758477 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2916a9b2-52c4-4a67-a41b-78691c711581 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:26.767153 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-2916a9b2-52c4-4a67-a41b-78691c711581 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:26.782920 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2916a9b2-52c4-4a67-a41b-78691c711581 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:26.788795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275aeef2-e846-4077-903b-778f1032b3ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:26.789527 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275aeef2-e846-4077-903b-778f1032b3ed admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:26.789934 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275aeef2-e846-4077-903b-778f1032b3ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:26.790350 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275aeef2-e846-4077-903b-778f1032b3ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:26.791142 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275aeef2-e846-4077-903b-778f1032b3ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:26.793107 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275aeef2-e846-4077-903b-778f1032b3ed admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:26.816616 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-f97b2371-0674-431c-9fdd-444cea05fb2e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a2c0b2a9bb104ef78ba29344336c7027'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:26.820021 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1022/2059] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:26 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:26.822247 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275aeef2-e846-4077-903b-778f1032b3ed admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:26.830992 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-f97b2371-0674-431c-9fdd-444cea05fb2e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a2c0b2a9bb104ef78ba29344336c7027'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:26.834681 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:26.849244 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275aeef2-e846-4077-903b-778f1032b3ed admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:26.849480 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-f97b2371-0674-431c-9fdd-444cea05fb2e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a2c0b2a9bb104ef78ba29344336c7027'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:26.850766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-f97b2371-0674-431c-9fdd-444cea05fb2e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a2c0b2a9bb104ef78ba29344336c7027'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:26.856607 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f98bc8-4b96-41a4-9e9d-2ccebc7f192c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:26.857513 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f98bc8-4b96-41a4-9e9d-2ccebc7f192c admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:26.857801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f98bc8-4b96-41a4-9e9d-2ccebc7f192c admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:26.858105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f98bc8-4b96-41a4-9e9d-2ccebc7f192c admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:26.858803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f98bc8-4b96-41a4-9e9d-2ccebc7f192c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:26.860953 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f98bc8-4b96-41a4-9e9d-2ccebc7f192c admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:26.862333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-f97b2371-0674-431c-9fdd-444cea05fb2e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a2c0b2a9bb104ef78ba29344336c7027'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:26.875846 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1038/2060] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:26 2022] DELETE /identity/v3/users/a2c0b2a9bb104ef78ba29344336c7027 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:26.889804 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1023/2061] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:26 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:26.894310 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97b2371-0674-431c-9fdd-444cea05fb2e admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:26.897628 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-f97b2371-0674-431c-9fdd-444cea05fb2e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:26.996541 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97b2371-0674-431c-9fdd-444cea05fb2e admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:27.001691 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa4e286-c463-4291-a7be-ddedb84b9089 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:27.002435 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa4e286-c463-4291-a7be-ddedb84b9089 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:27.002853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa4e286-c463-4291-a7be-ddedb84b9089 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:27.003218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa4e286-c463-4291-a7be-ddedb84b9089 admin admin] PATH_INFO: `/v3/projects/62001eb475994cac87565718b01f7d05/users/9ea890b6525a4ff5b5c54b81b48db659/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:27.046645 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa4e286-c463-4291-a7be-ddedb84b9089 admin admin] RBAC: Authorizing `identity:create_grant(project_id=62001eb475994cac87565718b01f7d05, user_id=9ea890b6525a4ff5b5c54b81b48db659, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:27.049456 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa4e286-c463-4291-a7be-ddedb84b9089 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:27.068397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1039/2062] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:26 2022] PUT /identity/v3/projects/62001eb475994cac87565718b01f7d05/users/9ea890b6525a4ff5b5c54b81b48db659/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:27.079094 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a3e305-8096-4d40-aeaa-fd128f8c96a4 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:27.079972 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a3e305-8096-4d40-aeaa-fd128f8c96a4 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:27.079972 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a3e305-8096-4d40-aeaa-fd128f8c96a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:27.108819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-84a3e305-8096-4d40-aeaa-fd128f8c96a4 None None] MFA Rules not processed for user `9ea890b6525a4ff5b5c54b81b48db659`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:27.109940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-84a3e305-8096-4d40-aeaa-fd128f8c96a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:27.178057 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:27.188780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1024/2063] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:27 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:27.193665 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa4e286-c463-4291-a7be-ddedb84b9089 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:27.252852 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a3e305-8096-4d40-aeaa-fd128f8c96a4 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:27.254329 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-84a3e305-8096-4d40-aeaa-fd128f8c96a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:27.264597 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa4e286-c463-4291-a7be-ddedb84b9089 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:27.269906 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca8e0cf-4fdb-4dda-b0f8-7f09b7702e1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:27.270519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a3e305-8096-4d40-aeaa-fd128f8c96a4 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:27.271354 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca8e0cf-4fdb-4dda-b0f8-7f09b7702e1d admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:27.272097 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca8e0cf-4fdb-4dda-b0f8-7f09b7702e1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:27.272549 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca8e0cf-4fdb-4dda-b0f8-7f09b7702e1d admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:27.274502 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca8e0cf-4fdb-4dda-b0f8-7f09b7702e1d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:27.276387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca8e0cf-4fdb-4dda-b0f8-7f09b7702e1d admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:27.279644 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859a9c82-ff8f-448b-907c-ccabed245235 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:27.280503 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859a9c82-ff8f-448b-907c-ccabed245235 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:27.280942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859a9c82-ff8f-448b-907c-ccabed245235 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:27.281216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859a9c82-ff8f-448b-907c-ccabed245235 admin admin] PATH_INFO: `/v3/projects/ba17f3156ae24556b81a64012a8d4cbc` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:27.282569 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859a9c82-ff8f-448b-907c-ccabed245235 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ba17f3156ae24556b81a64012a8d4cbc)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:27.284476 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859a9c82-ff8f-448b-907c-ccabed245235 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:27.314497 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1040/2064] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:27 2022] POST /identity/v3/projects => generated 373 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:27.315797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca8e0cf-4fdb-4dda-b0f8-7f09b7702e1d admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:27.338814 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-859a9c82-ff8f-448b-907c-ccabed245235 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ba17f3156ae24556b81a64012a8d4cbc', 'request_id': 'req-859a9c82-ff8f-448b-907c-ccabed245235'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:27.344649 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-859a9c82-ff8f-448b-907c-ccabed245235 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ba17f3156ae24556b81a64012a8d4cbc', 'request_id': 'req-859a9c82-ff8f-448b-907c-ccabed245235'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:27.353697 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-859a9c82-ff8f-448b-907c-ccabed245235 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ba17f3156ae24556b81a64012a8d4cbc', 'request_id': 'req-859a9c82-ff8f-448b-907c-ccabed245235'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:27.369603 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-859a9c82-ff8f-448b-907c-ccabed245235 admin admin] The token cache is being invalidate because project ba17f3156ae24556b81a64012a8d4cbc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:27.369603 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-859a9c82-ff8f-448b-907c-ccabed245235 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:27.370586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-859a9c82-ff8f-448b-907c-ccabed245235 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:27.372549 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1025/2065] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:27 2022] DELETE /identity/v3/projects/ba17f3156ae24556b81a64012a8d4cbc => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:27.375416 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:27.377590 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa36b14-8b35-4bae-a5f2-bf80677099a4 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:27.377881 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa36b14-8b35-4bae-a5f2-bf80677099a4 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:27.378151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa36b14-8b35-4bae-a5f2-bf80677099a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:27.393960 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca8e0cf-4fdb-4dda-b0f8-7f09b7702e1d admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:27.399750 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-5fa36b14-8b35-4bae-a5f2-bf80677099a4 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:27.400979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-5fa36b14-8b35-4bae-a5f2-bf80677099a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:27.457355 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147a1a07-9598-4211-a50e-7b45f91dc7f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:27.458002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147a1a07-9598-4211-a50e-7b45f91dc7f4 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:27.458921 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147a1a07-9598-4211-a50e-7b45f91dc7f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:27.459198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147a1a07-9598-4211-a50e-7b45f91dc7f4 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:27.459839 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147a1a07-9598-4211-a50e-7b45f91dc7f4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:27.461769 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147a1a07-9598-4211-a50e-7b45f91dc7f4 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:27.478414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1026/2066] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:27.485931 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1041/2067] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:27 2022] POST /identity/v3/users => generated 293 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:27.487207 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe00d927-86db-493b-ae90-b246173ec00f None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:27.487498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe00d927-86db-493b-ae90-b246173ec00f None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:27.487780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe00d927-86db-493b-ae90-b246173ec00f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:27.489804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147a1a07-9598-4211-a50e-7b45f91dc7f4 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:27.491442 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-147a1a07-9598-4211-a50e-7b45f91dc7f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:27.503292 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147a1a07-9598-4211-a50e-7b45f91dc7f4 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:27.508446 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b1831f-a303-4209-a4dc-e168222fb20f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:27.508849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-fe00d927-86db-493b-ae90-b246173ec00f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:27.509056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b1831f-a303-4209-a4dc-e168222fb20f admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:27.509337 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b1831f-a303-4209-a4dc-e168222fb20f admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:27.509614 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b1831f-a303-4209-a4dc-e168222fb20f admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:27.510088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-fe00d927-86db-493b-ae90-b246173ec00f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:27.510271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b1831f-a303-4209-a4dc-e168222fb20f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:27.511912 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b1831f-a303-4209-a4dc-e168222fb20f admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:27.518176 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1027/2068] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:58:27.522585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe00d927-86db-493b-ae90-b246173ec00f None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:27.524954 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1042/2069] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:27 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:58:27.529240 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b1831f-a303-4209-a4dc-e168222fb20f admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:27.532752 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe00d927-86db-493b-ae90-b246173ec00f None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:27.537832 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1d0f79-820f-4d46-80e4-fb23f16528f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:27.538447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1d0f79-820f-4d46-80e4-fb23f16528f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:27.538774 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1d0f79-820f-4d46-80e4-fb23f16528f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:27.539047 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1d0f79-820f-4d46-80e4-fb23f16528f8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:27.539732 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1d0f79-820f-4d46-80e4-fb23f16528f8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:58:27.540055 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1d0f79-820f-4d46-80e4-fb23f16528f8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:27.541703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1d0f79-820f-4d46-80e4-fb23f16528f8 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:27.542009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b1831f-a303-4209-a4dc-e168222fb20f admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:27.546959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b546b1-7522-49f1-8e6b-1ecd164565f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:27.547621 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b546b1-7522-49f1-8e6b-1ecd164565f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:27.547899 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b546b1-7522-49f1-8e6b-1ecd164565f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:27.548172 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b546b1-7522-49f1-8e6b-1ecd164565f3 admin admin] PATH_INFO: `/v3/projects/7ea26ba7c2d64a35a69fed875d1dda1c/users/3dfa576ddf2840c6aead9ae2c07f1df2/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:27.554108 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1028/2070] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:58:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:58:27.558510 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1d0f79-820f-4d46-80e4-fb23f16528f8 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:27.563953 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b546b1-7522-49f1-8e6b-1ecd164565f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7ea26ba7c2d64a35a69fed875d1dda1c, user_id=3dfa576ddf2840c6aead9ae2c07f1df2, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:27.567188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b546b1-7522-49f1-8e6b-1ecd164565f3 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:27.568692 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1d0f79-820f-4d46-80e4-fb23f16528f8 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:27.574728 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924adc03-2af1-4079-ba8d-71f8b8d33097 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:27.575529 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924adc03-2af1-4079-ba8d-71f8b8d33097 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:27.575871 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924adc03-2af1-4079-ba8d-71f8b8d33097 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:27.576148 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924adc03-2af1-4079-ba8d-71f8b8d33097 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:27.576803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924adc03-2af1-4079-ba8d-71f8b8d33097 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:27.578802 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924adc03-2af1-4079-ba8d-71f8b8d33097 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:27.582201 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1043/2071] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:27 2022] PUT /identity/v3/projects/7ea26ba7c2d64a35a69fed875d1dda1c/users/3dfa576ddf2840c6aead9ae2c07f1df2/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:27.586398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b546b1-7522-49f1-8e6b-1ecd164565f3 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:27.622436 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1029/2072] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:27 2022] POST /identity/v3/projects => generated 379 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:27.626979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924adc03-2af1-4079-ba8d-71f8b8d33097 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:27.665214 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b546b1-7522-49f1-8e6b-1ecd164565f3 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:27.675380 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:27.702002 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924adc03-2af1-4079-ba8d-71f8b8d33097 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:27.707543 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6beab4-9661-4bf0-9f73-642dd5711fb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:27.708884 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6beab4-9661-4bf0-9f73-642dd5711fb7 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:27.709219 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6beab4-9661-4bf0-9f73-642dd5711fb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:27.709492 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6beab4-9661-4bf0-9f73-642dd5711fb7 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:27.710151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6beab4-9661-4bf0-9f73-642dd5711fb7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:27.712081 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6beab4-9661-4bf0-9f73-642dd5711fb7 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:27.723691 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:27.739195 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d7f11b-f372-46c8-8c23-1948af35f6fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:27.739882 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d7f11b-f372-46c8-8c23-1948af35f6fb admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:27.740170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d7f11b-f372-46c8-8c23-1948af35f6fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:27.743701 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1030/2073] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:27 2022] POST /identity/v3/users => generated 290 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:58:27.743842 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d7f11b-f372-46c8-8c23-1948af35f6fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:27.743842 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d7f11b-f372-46c8-8c23-1948af35f6fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:27.743842 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d7f11b-f372-46c8-8c23-1948af35f6fb admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:27.744723 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6beab4-9661-4bf0-9f73-642dd5711fb7 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:27.753772 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1044/2074] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:27 2022] GET /identity/v3/roles => generated 1420 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:58:27.754717 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6beab4-9661-4bf0-9f73-642dd5711fb7 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:27.757720 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d7f11b-f372-46c8-8c23-1948af35f6fb admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:27.759778 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5179e29-c720-4e03-a120-1863030a169e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:27.760375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5179e29-c720-4e03-a120-1863030a169e admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:27.760706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5179e29-c720-4e03-a120-1863030a169e admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:27.760943 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5179e29-c720-4e03-a120-1863030a169e admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:27.761583 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5179e29-c720-4e03-a120-1863030a169e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:27.763204 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5179e29-c720-4e03-a120-1863030a169e admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:27.767615 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d7f11b-f372-46c8-8c23-1948af35f6fb admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:27.773435 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df312fed-f5bd-48e0-b16d-5f5c334b9881 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:27.774094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df312fed-f5bd-48e0-b16d-5f5c334b9881 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:27.774375 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df312fed-f5bd-48e0-b16d-5f5c334b9881 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:27.774680 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df312fed-f5bd-48e0-b16d-5f5c334b9881 admin admin] PATH_INFO: `/v3/projects/7ea26ba7c2d64a35a69fed875d1dda1c/users/3dfa576ddf2840c6aead9ae2c07f1df2/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:27.779591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1031/2075] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:27 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:27.779727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df312fed-f5bd-48e0-b16d-5f5c334b9881 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7ea26ba7c2d64a35a69fed875d1dda1c, user_id=3dfa576ddf2840c6aead9ae2c07f1df2, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:27.779727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df312fed-f5bd-48e0-b16d-5f5c334b9881 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:27.780918 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5179e29-c720-4e03-a120-1863030a169e admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:27.791640 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5179e29-c720-4e03-a120-1863030a169e admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:27.796257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1045/2076] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:27 2022] PUT /identity/v3/projects/7ea26ba7c2d64a35a69fed875d1dda1c/users/3dfa576ddf2840c6aead9ae2c07f1df2/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:27.807777 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c685186-a248-4960-aa05-31decce690ea None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:27.808289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c685186-a248-4960-aa05-31decce690ea None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:27.808667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c685186-a248-4960-aa05-31decce690ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:27.840118 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-5c685186-a248-4960-aa05-31decce690ea None None] MFA Rules not processed for user `3dfa576ddf2840c6aead9ae2c07f1df2`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:27.841633 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-5c685186-a248-4960-aa05-31decce690ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:27.854145 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78775c22-1a8a-451e-bba5-4d56df644902 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:27.854145 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78775c22-1a8a-451e-bba5-4d56df644902 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:27.854437 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78775c22-1a8a-451e-bba5-4d56df644902 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:27.854733 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78775c22-1a8a-451e-bba5-4d56df644902 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:27.855411 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78775c22-1a8a-451e-bba5-4d56df644902 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:27.856985 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78775c22-1a8a-451e-bba5-4d56df644902 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:27.867494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1032/2077] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:27 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:27.872341 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78775c22-1a8a-451e-bba5-4d56df644902 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:27.883304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78775c22-1a8a-451e-bba5-4d56df644902 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:27.890673 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0ed8b0-53b3-4691-8e1a-6ee4b47c0edb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:27.891353 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0ed8b0-53b3-4691-8e1a-6ee4b47c0edb admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:27.891631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0ed8b0-53b3-4691-8e1a-6ee4b47c0edb admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:27.892020 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0ed8b0-53b3-4691-8e1a-6ee4b47c0edb admin admin] PATH_INFO: `/v3/projects/a19138ec22f64bb998d60532ef5920bb/users/747a1d5508c14124b6bb358b8fbeb082/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:27.908567 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0ed8b0-53b3-4691-8e1a-6ee4b47c0edb admin admin] RBAC: Authorizing `identity:create_grant(project_id=a19138ec22f64bb998d60532ef5920bb, user_id=747a1d5508c14124b6bb358b8fbeb082, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:27.911370 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0ed8b0-53b3-4691-8e1a-6ee4b47c0edb admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:27.921627 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1046/2078] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:27 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:58:27.926254 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1033/2079] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:27 2022] PUT /identity/v3/projects/a19138ec22f64bb998d60532ef5920bb/users/747a1d5508c14124b6bb358b8fbeb082/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:27.935554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c685186-a248-4960-aa05-31decce690ea None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:27.936050 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-5c685186-a248-4960-aa05-31decce690ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:27.938168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9569935-ab3e-4ffe-bf18-39f2e9af486c None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:27.938539 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9569935-ab3e-4ffe-bf18-39f2e9af486c None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:27.938816 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9569935-ab3e-4ffe-bf18-39f2e9af486c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:27.970173 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-a9569935-ab3e-4ffe-bf18-39f2e9af486c None None] MFA Rules not processed for user `747a1d5508c14124b6bb358b8fbeb082`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:27.971853 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-a9569935-ab3e-4ffe-bf18-39f2e9af486c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:28.025240 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c685186-a248-4960-aa05-31decce690ea None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:28.096036 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1034/2080] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:27 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:58:28.100518 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9569935-ab3e-4ffe-bf18-39f2e9af486c None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:28.100866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7004e491-b845-4178-a29c-b371131a067b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:28.101685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7004e491-b845-4178-a29c-b371131a067b service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:28.102164 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7004e491-b845-4178-a29c-b371131a067b service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:28.102421 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7004e491-b845-4178-a29c-b371131a067b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:28.131869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:28.139055 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:28.201881 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9569935-ab3e-4ffe-bf18-39f2e9af486c None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:28.206140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7004e491-b845-4178-a29c-b371131a067b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:28.206940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81cfe53-fe44-4227-94f3-7403fe237668 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:28.207630 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7004e491-b845-4178-a29c-b371131a067b service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:28.207778 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81cfe53-fe44-4227-94f3-7403fe237668 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:28.208243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81cfe53-fe44-4227-94f3-7403fe237668 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:28.208592 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81cfe53-fe44-4227-94f3-7403fe237668 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:28.209348 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81cfe53-fe44-4227-94f3-7403fe237668 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:28.211611 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81cfe53-fe44-4227-94f3-7403fe237668 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:28.218327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1047/2081] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:27 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 285 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:28.248777 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1035/2082] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:28 2022] POST /identity/v3/projects => generated 381 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:58:28.255702 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7004e491-b845-4178-a29c-b371131a067b service cinder] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:28.270186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7004e491-b845-4178-a29c-b371131a067b service cinder] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:28.341599 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709a299f-8e70-40ec-833b-0463b7b18708 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:28.342228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709a299f-8e70-40ec-833b-0463b7b18708 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:28.342544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709a299f-8e70-40ec-833b-0463b7b18708 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:28.343200 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709a299f-8e70-40ec-833b-0463b7b18708 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:28.344006 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709a299f-8e70-40ec-833b-0463b7b18708 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:28.346081 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709a299f-8e70-40ec-833b-0463b7b18708 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:28.378501 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1048/2083] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:28 2022] POST /identity/v3/users => generated 297 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:28.381546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81cfe53-fe44-4227-94f3-7403fe237668 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:28.393765 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81cfe53-fe44-4227-94f3-7403fe237668 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:28.399744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f28cfc3-57b0-41ed-8864-c0059fdd3a5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:28.400514 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f28cfc3-57b0-41ed-8864-c0059fdd3a5d admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:28.400907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f28cfc3-57b0-41ed-8864-c0059fdd3a5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:28.401256 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f28cfc3-57b0-41ed-8864-c0059fdd3a5d admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:28.402035 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f28cfc3-57b0-41ed-8864-c0059fdd3a5d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:28.403854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f28cfc3-57b0-41ed-8864-c0059fdd3a5d admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:28.422738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1036/2084] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:28 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:28.422928 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709a299f-8e70-40ec-833b-0463b7b18708 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:28.433890 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709a299f-8e70-40ec-833b-0463b7b18708 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:28.440759 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bc2e48-d2f5-4e49-87d7-14f31aeb5096 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:28.443147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bc2e48-d2f5-4e49-87d7-14f31aeb5096 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:28.443455 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bc2e48-d2f5-4e49-87d7-14f31aeb5096 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:28.443710 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bc2e48-d2f5-4e49-87d7-14f31aeb5096 admin admin] PATH_INFO: `/v3/projects/136d4eecb3fe4340be435f8482545355/users/874e9c6f72794387bd426052e6d1e14d/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:28.463724 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3bc2e48-d2f5-4e49-87d7-14f31aeb5096 admin admin] RBAC: Authorizing `identity:create_grant(project_id=136d4eecb3fe4340be435f8482545355, user_id=874e9c6f72794387bd426052e6d1e14d, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:28.466867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3bc2e48-d2f5-4e49-87d7-14f31aeb5096 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:28.487026 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1049/2085] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:28 2022] PUT /identity/v3/projects/136d4eecb3fe4340be435f8482545355/users/874e9c6f72794387bd426052e6d1e14d/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:28.491434 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f28cfc3-57b0-41ed-8864-c0059fdd3a5d admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:28.511523 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f28cfc3-57b0-41ed-8864-c0059fdd3a5d admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:28.614744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709467d3-86b1-4619-9568-22eb12b58605 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:28.615548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709467d3-86b1-4619-9568-22eb12b58605 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:28.615884 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709467d3-86b1-4619-9568-22eb12b58605 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:28.616202 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709467d3-86b1-4619-9568-22eb12b58605 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:28.617192 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709467d3-86b1-4619-9568-22eb12b58605 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:28.618706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709467d3-86b1-4619-9568-22eb12b58605 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:28.635185 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1037/2086] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:28 2022] GET /identity/v3/roles => generated 1420 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:58:28.636913 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bc2e48-d2f5-4e49-87d7-14f31aeb5096 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:28.650908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bc2e48-d2f5-4e49-87d7-14f31aeb5096 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:28.657254 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d15aac9-e9e3-4030-84b0-9f9ae621a6cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:28.658019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d15aac9-e9e3-4030-84b0-9f9ae621a6cb admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:28.658414 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d15aac9-e9e3-4030-84b0-9f9ae621a6cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:28.658757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d15aac9-e9e3-4030-84b0-9f9ae621a6cb admin admin] PATH_INFO: `/v3/projects/136d4eecb3fe4340be435f8482545355/users/874e9c6f72794387bd426052e6d1e14d/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:28.660759 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d15aac9-e9e3-4030-84b0-9f9ae621a6cb admin admin] RBAC: Authorizing `identity:create_grant(project_id=136d4eecb3fe4340be435f8482545355, user_id=874e9c6f72794387bd426052e6d1e14d, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:28.663606 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d15aac9-e9e3-4030-84b0-9f9ae621a6cb admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:28.679496 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1050/2087] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:28 2022] PUT /identity/v3/projects/136d4eecb3fe4340be435f8482545355/users/874e9c6f72794387bd426052e6d1e14d/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:28.688155 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd326a6-cb7b-4524-9b76-064bdd64c58d None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:28.688438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd326a6-cb7b-4524-9b76-064bdd64c58d None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:28.688722 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd326a6-cb7b-4524-9b76-064bdd64c58d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:28.723982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-dbd326a6-cb7b-4524-9b76-064bdd64c58d None None] MFA Rules not processed for user `874e9c6f72794387bd426052e6d1e14d`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:28.725379 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-dbd326a6-cb7b-4524-9b76-064bdd64c58d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:28.754141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d15aac9-e9e3-4030-84b0-9f9ae621a6cb admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:28.801530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1038/2088] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:28 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:28.813040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd326a6-cb7b-4524-9b76-064bdd64c58d None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:28.814370 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-dbd326a6-cb7b-4524-9b76-064bdd64c58d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:28.821487 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d15aac9-e9e3-4030-84b0-9f9ae621a6cb admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:28.828826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4ff1c9-96bb-474a-94a6-d73e55d8c1df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:28.829635 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4ff1c9-96bb-474a-94a6-d73e55d8c1df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:28.829900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4ff1c9-96bb-474a-94a6-d73e55d8c1df admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:28.830194 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4ff1c9-96bb-474a-94a6-d73e55d8c1df admin admin] PATH_INFO: `/v3/users/9ea890b6525a4ff5b5c54b81b48db659` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:28.831558 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:28.831777 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4ff1c9-96bb-474a-94a6-d73e55d8c1df admin admin] RBAC: Authorizing `identity:delete_user(user_id=9ea890b6525a4ff5b5c54b81b48db659)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:28.834548 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4ff1c9-96bb-474a-94a6-d73e55d8c1df admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:28.857367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=93,keystone:INSERT=5,keystone:DELETE=2 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:28.881648 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd326a6-cb7b-4524-9b76-064bdd64c58d None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:28.885326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbeb85d5-fea2-4380-82ba-1c076462d48a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:28.885939 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbeb85d5-fea2-4380-82ba-1c076462d48a service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:28.886217 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbeb85d5-fea2-4380-82ba-1c076462d48a service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:28.886515 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbeb85d5-fea2-4380-82ba-1c076462d48a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:28.886751 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-4f4ff1c9-96bb-474a-94a6-d73e55d8c1df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9ea890b6525a4ff5b5c54b81b48db659'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:28.894818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-4f4ff1c9-96bb-474a-94a6-d73e55d8c1df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9ea890b6525a4ff5b5c54b81b48db659'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:28.898063 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbeb85d5-fea2-4380-82ba-1c076462d48a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:28.899547 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbeb85d5-fea2-4380-82ba-1c076462d48a service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:28.907181 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-4f4ff1c9-96bb-474a-94a6-d73e55d8c1df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9ea890b6525a4ff5b5c54b81b48db659'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:28.907884 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-4f4ff1c9-96bb-474a-94a6-d73e55d8c1df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9ea890b6525a4ff5b5c54b81b48db659'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:28.909069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1039/2089] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:28 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:58:28.915562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-4f4ff1c9-96bb-474a-94a6-d73e55d8c1df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9ea890b6525a4ff5b5c54b81b48db659'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:28.922420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1051/2090] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:28 2022] DELETE /identity/v3/users/9ea890b6525a4ff5b5c54b81b48db659 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:28.927631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e899ab-f809-4d19-84d0-3b2043900ec5 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:28.927927 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e899ab-f809-4d19-84d0-3b2043900ec5 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:28.928196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e899ab-f809-4d19-84d0-3b2043900ec5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:28.948198 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-a5e899ab-f809-4d19-84d0-3b2043900ec5 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:28.950075 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-a5e899ab-f809-4d19-84d0-3b2043900ec5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:29.018864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1040/2091] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:29.029386 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4ff1c9-96bb-474a-94a6-d73e55d8c1df admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:29.030815 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-4f4ff1c9-96bb-474a-94a6-d73e55d8c1df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:29.102347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4ff1c9-96bb-474a-94a6-d73e55d8c1df admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:29.106914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5454ad9d-3e4b-46b5-83c9-cf7ad7618fa3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:29.107352 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5454ad9d-3e4b-46b5-83c9-cf7ad7618fa3 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:29.107777 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5454ad9d-3e4b-46b5-83c9-cf7ad7618fa3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:29.108054 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5454ad9d-3e4b-46b5-83c9-cf7ad7618fa3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:29.119539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5454ad9d-3e4b-46b5-83c9-cf7ad7618fa3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:29.121010 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5454ad9d-3e4b-46b5-83c9-cf7ad7618fa3 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:29.145513 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1052/2092] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:29.385266 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e899ab-f809-4d19-84d0-3b2043900ec5 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:29.387300 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-a5e899ab-f809-4d19-84d0-3b2043900ec5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:29.401673 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e899ab-f809-4d19-84d0-3b2043900ec5 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:29.411124 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da6db25-e888-4723-b396-ced8d0bad0b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:29.411904 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da6db25-e888-4723-b396-ced8d0bad0b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:29.412276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da6db25-e888-4723-b396-ced8d0bad0b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:29.412675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da6db25-e888-4723-b396-ced8d0bad0b1 admin admin] PATH_INFO: `/v3/users/747a1d5508c14124b6bb358b8fbeb082` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:29.413996 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da6db25-e888-4723-b396-ced8d0bad0b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=747a1d5508c14124b6bb358b8fbeb082)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:29.415927 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da6db25-e888-4723-b396-ced8d0bad0b1 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:29.482027 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-5da6db25-e888-4723-b396-ced8d0bad0b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '747a1d5508c14124b6bb358b8fbeb082'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:29.491055 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-5da6db25-e888-4723-b396-ced8d0bad0b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '747a1d5508c14124b6bb358b8fbeb082'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:29.491683 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-5da6db25-e888-4723-b396-ced8d0bad0b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '747a1d5508c14124b6bb358b8fbeb082'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:29.492317 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-5da6db25-e888-4723-b396-ced8d0bad0b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '747a1d5508c14124b6bb358b8fbeb082'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:29.500530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-5da6db25-e888-4723-b396-ced8d0bad0b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '747a1d5508c14124b6bb358b8fbeb082'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:29.520812 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1041/2093] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:29 2022] DELETE /identity/v3/users/747a1d5508c14124b6bb358b8fbeb082 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:29.525751 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb9457a-01f6-418b-9578-8ecbb511bb25 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:29.526365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb9457a-01f6-418b-9578-8ecbb511bb25 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:29.526855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb9457a-01f6-418b-9578-8ecbb511bb25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:29.549543 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da6db25-e888-4723-b396-ced8d0bad0b1 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:29.551016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-5da6db25-e888-4723-b396-ced8d0bad0b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:29.553472 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-ebb9457a-01f6-418b-9578-8ecbb511bb25 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:29.554851 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-ebb9457a-01f6-418b-9578-8ecbb511bb25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:29.598392 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:29.629656 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1053/2094] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:29.641224 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb9457a-01f6-418b-9578-8ecbb511bb25 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:29.641422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da6db25-e888-4723-b396-ced8d0bad0b1 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:29.642902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-ebb9457a-01f6-418b-9578-8ecbb511bb25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:29.651086 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c57faa-36a9-474d-921d-5ba4ded34690 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:29.651774 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c57faa-36a9-474d-921d-5ba4ded34690 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:29.652055 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c57faa-36a9-474d-921d-5ba4ded34690 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:29.652327 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c57faa-36a9-474d-921d-5ba4ded34690 admin admin] PATH_INFO: `/v3/projects/62001eb475994cac87565718b01f7d05` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:29.653532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c57faa-36a9-474d-921d-5ba4ded34690 admin admin] RBAC: Authorizing `identity:delete_project(project_id=62001eb475994cac87565718b01f7d05)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:29.656662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c57faa-36a9-474d-921d-5ba4ded34690 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:29.701656 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:29.717497 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b5c57faa-36a9-474d-921d-5ba4ded34690 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '62001eb475994cac87565718b01f7d05', 'request_id': 'req-b5c57faa-36a9-474d-921d-5ba4ded34690'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:29.723817 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb9457a-01f6-418b-9578-8ecbb511bb25 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:29.724127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b5c57faa-36a9-474d-921d-5ba4ded34690 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '62001eb475994cac87565718b01f7d05', 'request_id': 'req-b5c57faa-36a9-474d-921d-5ba4ded34690'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:29.733681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b5c57faa-36a9-474d-921d-5ba4ded34690 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '62001eb475994cac87565718b01f7d05', 'request_id': 'req-b5c57faa-36a9-474d-921d-5ba4ded34690'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:29.750270 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b5c57faa-36a9-474d-921d-5ba4ded34690 admin admin] The token cache is being invalidate because project 62001eb475994cac87565718b01f7d05 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:29.750918 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b5c57faa-36a9-474d-921d-5ba4ded34690 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:29.751486 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b5c57faa-36a9-474d-921d-5ba4ded34690 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:29.753369 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1042/2095] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:29 2022] DELETE /identity/v3/projects/62001eb475994cac87565718b01f7d05 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:29.758948 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c57faa-36a9-474d-921d-5ba4ded34690 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:29.760694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-b5c57faa-36a9-474d-921d-5ba4ded34690 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:29.793125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b689e2-0a6b-43be-9f2c-69c30523fe99 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:29.793719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b689e2-0a6b-43be-9f2c-69c30523fe99 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:29.794054 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b689e2-0a6b-43be-9f2c-69c30523fe99 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:29.794604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b689e2-0a6b-43be-9f2c-69c30523fe99 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:29.797624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-65b689e2-0a6b-43be-9f2c-69c30523fe99 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:29.866964 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c57faa-36a9-474d-921d-5ba4ded34690 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:29.875602 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26149069-628d-4076-9ca2-b4fc1c4c95b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:29.876399 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26149069-628d-4076-9ca2-b4fc1c4c95b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:29.876548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26149069-628d-4076-9ca2-b4fc1c4c95b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:29.876827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26149069-628d-4076-9ca2-b4fc1c4c95b3 admin admin] PATH_INFO: `/v3/users/3dfa576ddf2840c6aead9ae2c07f1df2` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:29.877369 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b689e2-0a6b-43be-9f2c-69c30523fe99 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:29.879337 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b689e2-0a6b-43be-9f2c-69c30523fe99 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:29.879697 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26149069-628d-4076-9ca2-b4fc1c4c95b3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3dfa576ddf2840c6aead9ae2c07f1df2)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:29.882606 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26149069-628d-4076-9ca2-b4fc1c4c95b3 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:29.903490 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1054/2096] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 263 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:58:29.940009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-26149069-628d-4076-9ca2-b4fc1c4c95b3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3dfa576ddf2840c6aead9ae2c07f1df2'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:29.948681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-26149069-628d-4076-9ca2-b4fc1c4c95b3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3dfa576ddf2840c6aead9ae2c07f1df2'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:29.949328 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-26149069-628d-4076-9ca2-b4fc1c4c95b3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3dfa576ddf2840c6aead9ae2c07f1df2'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:29.949982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-26149069-628d-4076-9ca2-b4fc1c4c95b3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3dfa576ddf2840c6aead9ae2c07f1df2'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:29.956334 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:29.960324 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-26149069-628d-4076-9ca2-b4fc1c4c95b3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3dfa576ddf2840c6aead9ae2c07f1df2'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:29.982596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1043/2097] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:29 2022] DELETE /identity/v3/users/3dfa576ddf2840c6aead9ae2c07f1df2 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:30.218978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b689e2-0a6b-43be-9f2c-69c30523fe99 service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:30.226270 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-65b689e2-0a6b-43be-9f2c-69c30523fe99 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:30.243769 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b689e2-0a6b-43be-9f2c-69c30523fe99 service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:30.309416 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0739a087-7e21-4a45-b86a-778a153fc66b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:30.310037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0739a087-7e21-4a45-b86a-778a153fc66b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:30.310324 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0739a087-7e21-4a45-b86a-778a153fc66b admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:30.310601 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0739a087-7e21-4a45-b86a-778a153fc66b admin admin] PATH_INFO: `/v3/projects/a19138ec22f64bb998d60532ef5920bb` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:30.311729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0739a087-7e21-4a45-b86a-778a153fc66b admin admin] RBAC: Authorizing `identity:delete_project(project_id=a19138ec22f64bb998d60532ef5920bb)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:30.313670 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0739a087-7e21-4a45-b86a-778a153fc66b admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:30.320832 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26149069-628d-4076-9ca2-b4fc1c4c95b3 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:30.322270 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-26149069-628d-4076-9ca2-b4fc1c4c95b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:30.355877 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26149069-628d-4076-9ca2-b4fc1c4c95b3 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:30.368226 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:30.369968 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-0739a087-7e21-4a45-b86a-778a153fc66b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a19138ec22f64bb998d60532ef5920bb', 'request_id': 'req-0739a087-7e21-4a45-b86a-778a153fc66b'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:30.377454 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-0739a087-7e21-4a45-b86a-778a153fc66b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a19138ec22f64bb998d60532ef5920bb', 'request_id': 'req-0739a087-7e21-4a45-b86a-778a153fc66b'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:30.385763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-0739a087-7e21-4a45-b86a-778a153fc66b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a19138ec22f64bb998d60532ef5920bb', 'request_id': 'req-0739a087-7e21-4a45-b86a-778a153fc66b'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:30.398342 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-0739a087-7e21-4a45-b86a-778a153fc66b admin admin] The token cache is being invalidate because project a19138ec22f64bb998d60532ef5920bb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:30.399009 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-0739a087-7e21-4a45-b86a-778a153fc66b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:30.399631 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-0739a087-7e21-4a45-b86a-778a153fc66b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:30.401764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1055/2098] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:30 2022] DELETE /identity/v3/projects/a19138ec22f64bb998d60532ef5920bb => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:30.406094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0739a087-7e21-4a45-b86a-778a153fc66b admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:30.407708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-0739a087-7e21-4a45-b86a-778a153fc66b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:30.427090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad4c45d-ee98-4f06-97c6-1c48432373ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:30.427734 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad4c45d-ee98-4f06-97c6-1c48432373ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:30.428019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad4c45d-ee98-4f06-97c6-1c48432373ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:30.428292 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad4c45d-ee98-4f06-97c6-1c48432373ab admin admin] PATH_INFO: `/v3/projects/7ea26ba7c2d64a35a69fed875d1dda1c` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:30.429507 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad4c45d-ee98-4f06-97c6-1c48432373ab admin admin] RBAC: Authorizing `identity:delete_project(project_id=7ea26ba7c2d64a35a69fed875d1dda1c)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:30.431544 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad4c45d-ee98-4f06-97c6-1c48432373ab admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:30.483802 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0739a087-7e21-4a45-b86a-778a153fc66b admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:30.494249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-dad4c45d-ee98-4f06-97c6-1c48432373ab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7ea26ba7c2d64a35a69fed875d1dda1c', 'request_id': 'req-dad4c45d-ee98-4f06-97c6-1c48432373ab'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:30.500313 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-dad4c45d-ee98-4f06-97c6-1c48432373ab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7ea26ba7c2d64a35a69fed875d1dda1c', 'request_id': 'req-dad4c45d-ee98-4f06-97c6-1c48432373ab'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:30.515412 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-dad4c45d-ee98-4f06-97c6-1c48432373ab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7ea26ba7c2d64a35a69fed875d1dda1c', 'request_id': 'req-dad4c45d-ee98-4f06-97c6-1c48432373ab'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:30.539781 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-dad4c45d-ee98-4f06-97c6-1c48432373ab admin admin] The token cache is being invalidate because project 7ea26ba7c2d64a35a69fed875d1dda1c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:30.540071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-dad4c45d-ee98-4f06-97c6-1c48432373ab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:30.541382 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-dad4c45d-ee98-4f06-97c6-1c48432373ab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:30.543557 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1044/2099] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:30 2022] DELETE /identity/v3/projects/7ea26ba7c2d64a35a69fed875d1dda1c => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:30.548307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0d18c6-ec09-467f-8f99-a9d5858b6262 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:30.548591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0d18c6-ec09-467f-8f99-a9d5858b6262 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:30.548866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0d18c6-ec09-467f-8f99-a9d5858b6262 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:30.568741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2b61ea-a61e-4bb7-9e8d-07486e322f14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:30.570034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2b61ea-a61e-4bb7-9e8d-07486e322f14 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:30.570877 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2b61ea-a61e-4bb7-9e8d-07486e322f14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:30.571380 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2b61ea-a61e-4bb7-9e8d-07486e322f14 admin admin] PATH_INFO: `/v3/users/874e9c6f72794387bd426052e6d1e14d` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:30.571778 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-ae0d18c6-ec09-467f-8f99-a9d5858b6262 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:30.573259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2b61ea-a61e-4bb7-9e8d-07486e322f14 admin admin] RBAC: Authorizing `identity:delete_user(user_id=874e9c6f72794387bd426052e6d1e14d)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:30.573403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ae0d18c6-ec09-467f-8f99-a9d5858b6262 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:30.575425 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2b61ea-a61e-4bb7-9e8d-07486e322f14 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:30.586790 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1045/2100] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:30.594484 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c069c4ae-f8b7-4e0e-9e36-cab9b1edfe9b None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:30.594840 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c069c4ae-f8b7-4e0e-9e36-cab9b1edfe9b None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:30.595120 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c069c4ae-f8b7-4e0e-9e36-cab9b1edfe9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:30.619003 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-c069c4ae-f8b7-4e0e-9e36-cab9b1edfe9b None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:30.620378 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c069c4ae-f8b7-4e0e-9e36-cab9b1edfe9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:30.638707 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1046/2101] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:58:30.642776 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c069c4ae-f8b7-4e0e-9e36-cab9b1edfe9b None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:30.645229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-0e2b61ea-a61e-4bb7-9e8d-07486e322f14 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '874e9c6f72794387bd426052e6d1e14d'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:30.652495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c069c4ae-f8b7-4e0e-9e36-cab9b1edfe9b None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:30.654503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-0e2b61ea-a61e-4bb7-9e8d-07486e322f14 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '874e9c6f72794387bd426052e6d1e14d'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:30.657714 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08818ea9-0c28-41fb-9ba1-e1fcffa91511 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:30.658316 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08818ea9-0c28-41fb-9ba1-e1fcffa91511 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:30.658647 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08818ea9-0c28-41fb-9ba1-e1fcffa91511 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:30.658920 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08818ea9-0c28-41fb-9ba1-e1fcffa91511 admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:30.659568 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08818ea9-0c28-41fb-9ba1-e1fcffa91511 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:58:30.659868 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08818ea9-0c28-41fb-9ba1-e1fcffa91511 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:30.661986 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08818ea9-0c28-41fb-9ba1-e1fcffa91511 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:30.668317 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-0e2b61ea-a61e-4bb7-9e8d-07486e322f14 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '874e9c6f72794387bd426052e6d1e14d'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:30.669122 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-0e2b61ea-a61e-4bb7-9e8d-07486e322f14 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '874e9c6f72794387bd426052e6d1e14d'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:30.674270 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1047/2102] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:58:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:58:30.676586 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-0e2b61ea-a61e-4bb7-9e8d-07486e322f14 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '874e9c6f72794387bd426052e6d1e14d'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:30.679197 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08818ea9-0c28-41fb-9ba1-e1fcffa91511 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:30.681122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-08818ea9-0c28-41fb-9ba1-e1fcffa91511 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:30.689894 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1056/2103] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:30 2022] DELETE /identity/v3/users/874e9c6f72794387bd426052e6d1e14d => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:30.695837 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08818ea9-0c28-41fb-9ba1-e1fcffa91511 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:30.748046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:30.769529 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b2beff-0702-405d-bf6c-9919dc78379f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:30.770245 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b2beff-0702-405d-bf6c-9919dc78379f admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:30.771089 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b2beff-0702-405d-bf6c-9919dc78379f admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:30.771433 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b2beff-0702-405d-bf6c-9919dc78379f admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:30.772350 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b2beff-0702-405d-bf6c-9919dc78379f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:30.774698 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b2beff-0702-405d-bf6c-9919dc78379f admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:30.814986 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1048/2104] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:30 2022] POST /identity/v3/projects => generated 375 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:30.819366 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2b61ea-a61e-4bb7-9e8d-07486e322f14 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:30.825063 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:30.835991 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2b61ea-a61e-4bb7-9e8d-07486e322f14 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:30.905747 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6578a4e7-955e-4c6e-945f-ae7001401cfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:30.906276 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6578a4e7-955e-4c6e-945f-ae7001401cfa admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:30.906589 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6578a4e7-955e-4c6e-945f-ae7001401cfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:30.906868 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6578a4e7-955e-4c6e-945f-ae7001401cfa admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:30.907515 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6578a4e7-955e-4c6e-945f-ae7001401cfa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:30.909519 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6578a4e7-955e-4c6e-945f-ae7001401cfa admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:30.942006 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1057/2105] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:30 2022] POST /identity/v3/users => generated 288 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:58:30.945945 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b2beff-0702-405d-bf6c-9919dc78379f admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:30.957405 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b2beff-0702-405d-bf6c-9919dc78379f admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:30.963327 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0441660a-29dd-4265-96bf-a00d2898f5e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:30.963936 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0441660a-29dd-4265-96bf-a00d2898f5e0 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:30.964213 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0441660a-29dd-4265-96bf-a00d2898f5e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:30.964515 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0441660a-29dd-4265-96bf-a00d2898f5e0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:30.965148 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0441660a-29dd-4265-96bf-a00d2898f5e0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:30.966807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0441660a-29dd-4265-96bf-a00d2898f5e0 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:30.977530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1049/2106] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:30 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:30.981537 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6578a4e7-955e-4c6e-945f-ae7001401cfa admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:30.993803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6578a4e7-955e-4c6e-945f-ae7001401cfa admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:30.998867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb86e36-c34c-4299-b825-db744d81f668 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:30.999478 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb86e36-c34c-4299-b825-db744d81f668 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:30.999770 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb86e36-c34c-4299-b825-db744d81f668 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:31.000067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb86e36-c34c-4299-b825-db744d81f668 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:31.000690 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb86e36-c34c-4299-b825-db744d81f668 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:31.002287 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb86e36-c34c-4299-b825-db744d81f668 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:31.017496 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1058/2107] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:30 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:58:31.020348 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0441660a-29dd-4265-96bf-a00d2898f5e0 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:31.023079 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-0441660a-29dd-4265-96bf-a00d2898f5e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:31.027412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb86e36-c34c-4299-b825-db744d81f668 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:31.039517 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb86e36-c34c-4299-b825-db744d81f668 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:31.044364 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647dc8cd-9c57-4f95-b5d4-b0f45fbe61e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:31.045027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647dc8cd-9c57-4f95-b5d4-b0f45fbe61e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:31.046230 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647dc8cd-9c57-4f95-b5d4-b0f45fbe61e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:31.046230 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647dc8cd-9c57-4f95-b5d4-b0f45fbe61e3 admin admin] PATH_INFO: `/v3/projects/ed3423e6243e4316bd1e95aea4d4a531/users/b274269a7d2a4507b9d08615091d557f/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:31.046502 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0441660a-29dd-4265-96bf-a00d2898f5e0 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:31.051113 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd17043-86d8-4eda-ad2f-6b60d1886d12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:31.051878 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd17043-86d8-4eda-ad2f-6b60d1886d12 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:31.052148 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd17043-86d8-4eda-ad2f-6b60d1886d12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:31.053185 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd17043-86d8-4eda-ad2f-6b60d1886d12 admin admin] PATH_INFO: `/v3/projects/136d4eecb3fe4340be435f8482545355` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:31.054728 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd17043-86d8-4eda-ad2f-6b60d1886d12 admin admin] RBAC: Authorizing `identity:delete_project(project_id=136d4eecb3fe4340be435f8482545355)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:31.056673 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd17043-86d8-4eda-ad2f-6b60d1886d12 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:31.064656 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-647dc8cd-9c57-4f95-b5d4-b0f45fbe61e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ed3423e6243e4316bd1e95aea4d4a531, user_id=b274269a7d2a4507b9d08615091d557f, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:31.068085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-647dc8cd-9c57-4f95-b5d4-b0f45fbe61e3 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:31.081792 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1059/2108] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:31 2022] PUT /identity/v3/projects/ed3423e6243e4316bd1e95aea4d4a531/users/b274269a7d2a4507b9d08615091d557f/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:31.092178 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfec8b4-0b60-4739-8045-ebc58b0e128a None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:31.092521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfec8b4-0b60-4739-8045-ebc58b0e128a None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:31.092805 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfec8b4-0b60-4739-8045-ebc58b0e128a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:31.108873 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-3bd17043-86d8-4eda-ad2f-6b60d1886d12 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '136d4eecb3fe4340be435f8482545355', 'request_id': 'req-3bd17043-86d8-4eda-ad2f-6b60d1886d12'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:31.115422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-3bd17043-86d8-4eda-ad2f-6b60d1886d12 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '136d4eecb3fe4340be435f8482545355', 'request_id': 'req-3bd17043-86d8-4eda-ad2f-6b60d1886d12'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:31.122863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-1dfec8b4-0b60-4739-8045-ebc58b0e128a None None] MFA Rules not processed for user `b274269a7d2a4507b9d08615091d557f`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:31.123719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-3bd17043-86d8-4eda-ad2f-6b60d1886d12 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '136d4eecb3fe4340be435f8482545355', 'request_id': 'req-3bd17043-86d8-4eda-ad2f-6b60d1886d12'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:31.124084 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-1dfec8b4-0b60-4739-8045-ebc58b0e128a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:31.136223 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-3bd17043-86d8-4eda-ad2f-6b60d1886d12 admin admin] The token cache is being invalidate because project 136d4eecb3fe4340be435f8482545355 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:31.136515 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-3bd17043-86d8-4eda-ad2f-6b60d1886d12 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:31.137177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-3bd17043-86d8-4eda-ad2f-6b60d1886d12 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:31.138688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1050/2109] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:31 2022] DELETE /identity/v3/projects/136d4eecb3fe4340be435f8482545355 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:31.143351 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e85ace-f1bb-413a-8e04-c0ba47d9c64a None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:31.143533 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e85ace-f1bb-413a-8e04-c0ba47d9c64a None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:31.143802 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e85ace-f1bb-413a-8e04-c0ba47d9c64a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:31.164780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-01e85ace-f1bb-413a-8e04-c0ba47d9c64a None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:31.165941 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-01e85ace-f1bb-413a-8e04-c0ba47d9c64a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:31.197318 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1060/2110] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:31 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:58:31.204048 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfec8b4-0b60-4739-8045-ebc58b0e128a None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:31.205947 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-1dfec8b4-0b60-4739-8045-ebc58b0e128a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:31.236372 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1051/2111] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:31.244519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de14e68-a83f-42ef-8f8f-87d74a04a55f None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:31.244961 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de14e68-a83f-42ef-8f8f-87d74a04a55f None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:31.245221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de14e68-a83f-42ef-8f8f-87d74a04a55f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:31.268743 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-6de14e68-a83f-42ef-8f8f-87d74a04a55f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:31.269979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-6de14e68-a83f-42ef-8f8f-87d74a04a55f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:31.271842 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:31.279530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1052/2112] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:31.284523 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de14e68-a83f-42ef-8f8f-87d74a04a55f None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:31.291750 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfec8b4-0b60-4739-8045-ebc58b0e128a None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:31.293847 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de14e68-a83f-42ef-8f8f-87d74a04a55f None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:31.296483 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b72d05c-92bb-428c-b9d1-5fd33cacdcc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:31.297098 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b72d05c-92bb-428c-b9d1-5fd33cacdcc5 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:31.297377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b72d05c-92bb-428c-b9d1-5fd33cacdcc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:31.297651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b72d05c-92bb-428c-b9d1-5fd33cacdcc5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:31.298281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b72d05c-92bb-428c-b9d1-5fd33cacdcc5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:31.298569 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a789dec-a8d3-4b43-9ad3-874a70aff3c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:31.299181 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a789dec-a8d3-4b43-9ad3-874a70aff3c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:31.299460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a789dec-a8d3-4b43-9ad3-874a70aff3c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:31.299731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a789dec-a8d3-4b43-9ad3-874a70aff3c3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:31.300441 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a789dec-a8d3-4b43-9ad3-874a70aff3c3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:58:31.300533 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b72d05c-92bb-428c-b9d1-5fd33cacdcc5 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:31.300747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a789dec-a8d3-4b43-9ad3-874a70aff3c3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:31.302388 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a789dec-a8d3-4b43-9ad3-874a70aff3c3 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:31.320414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1053/2113] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:58:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:58:31.321623 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a789dec-a8d3-4b43-9ad3-874a70aff3c3 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:31.333678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1061/2114] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:31 2022] POST /identity/v3/projects => generated 377 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:58:31.333866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a789dec-a8d3-4b43-9ad3-874a70aff3c3 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:31.334894 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b72d05c-92bb-428c-b9d1-5fd33cacdcc5 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:31.356002 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:31.402524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd93abc-b3ad-4705-8a7f-31f32039c51b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:31.403126 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd93abc-b3ad-4705-8a7f-31f32039c51b admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:31.403401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd93abc-b3ad-4705-8a7f-31f32039c51b admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:31.403684 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd93abc-b3ad-4705-8a7f-31f32039c51b admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:31.404316 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd93abc-b3ad-4705-8a7f-31f32039c51b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:31.406231 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd93abc-b3ad-4705-8a7f-31f32039c51b admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:31.411196 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b72d05c-92bb-428c-b9d1-5fd33cacdcc5 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:31.415887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d204666c-c6cf-4fcb-ade7-d2eaf382ea8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:31.416409 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d204666c-c6cf-4fcb-ade7-d2eaf382ea8b admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:31.416682 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d204666c-c6cf-4fcb-ade7-d2eaf382ea8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:31.416951 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d204666c-c6cf-4fcb-ade7-d2eaf382ea8b admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:31.417582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d204666c-c6cf-4fcb-ade7-d2eaf382ea8b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:31.419570 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d204666c-c6cf-4fcb-ade7-d2eaf382ea8b admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:31.440839 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1054/2115] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:31 2022] POST /identity/v3/projects => generated 375 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:58:31.444640 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1062/2116] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:31 2022] POST /identity/v3/users => generated 295 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:58:31.446274 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd93abc-b3ad-4705-8a7f-31f32039c51b admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:31.449277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d204666c-c6cf-4fcb-ade7-d2eaf382ea8b admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:31.460579 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d204666c-c6cf-4fcb-ade7-d2eaf382ea8b admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:31.517332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd93abc-b3ad-4705-8a7f-31f32039c51b admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:31.523201 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf01022-309b-4c18-ac96-051534e4b6ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:31.523869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf01022-309b-4c18-ac96-051534e4b6ab admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:31.524100 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf01022-309b-4c18-ac96-051534e4b6ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:31.524369 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf01022-309b-4c18-ac96-051534e4b6ab admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:31.525008 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf01022-309b-4c18-ac96-051534e4b6ab admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:31.525268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d8915e-4e86-44d1-b109-27e2d09a2db7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:31.525961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d8915e-4e86-44d1-b109-27e2d09a2db7 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:31.526157 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d8915e-4e86-44d1-b109-27e2d09a2db7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:31.526601 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d8915e-4e86-44d1-b109-27e2d09a2db7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:31.527020 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf01022-309b-4c18-ac96-051534e4b6ab admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:31.527854 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d8915e-4e86-44d1-b109-27e2d09a2db7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:31.528953 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d8915e-4e86-44d1-b109-27e2d09a2db7 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:31.540072 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1063/2117] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:31 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:58:31.544972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d8915e-4e86-44d1-b109-27e2d09a2db7 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:31.553479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1055/2118] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:31 2022] POST /identity/v3/users => generated 288 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:31.557524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf01022-309b-4c18-ac96-051534e4b6ab admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:31.557975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d8915e-4e86-44d1-b109-27e2d09a2db7 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:31.562620 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3371ba-d290-431b-aadf-64b23e538f91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:31.563280 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3371ba-d290-431b-aadf-64b23e538f91 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:31.563555 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3371ba-d290-431b-aadf-64b23e538f91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:31.563829 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3371ba-d290-431b-aadf-64b23e538f91 admin admin] PATH_INFO: `/v3/projects/2ef37f3c7d514ccc96ad78d4ed7c4f6c/users/3ba140fc89594c2b89be8d73f9b1d8f6/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:31.569657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf01022-309b-4c18-ac96-051534e4b6ab admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:31.575321 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852442b6-a785-4b71-92d3-d5f284e92276 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:31.575928 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852442b6-a785-4b71-92d3-d5f284e92276 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:31.576230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852442b6-a785-4b71-92d3-d5f284e92276 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:31.576556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852442b6-a785-4b71-92d3-d5f284e92276 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:31.577213 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852442b6-a785-4b71-92d3-d5f284e92276 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:31.578875 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852442b6-a785-4b71-92d3-d5f284e92276 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:31.580582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3371ba-d290-431b-aadf-64b23e538f91 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2ef37f3c7d514ccc96ad78d4ed7c4f6c, user_id=3ba140fc89594c2b89be8d73f9b1d8f6, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:31.584051 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3371ba-d290-431b-aadf-64b23e538f91 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:31.589972 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1056/2119] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:31 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:31.594030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852442b6-a785-4b71-92d3-d5f284e92276 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:31.597764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1064/2120] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:31 2022] PUT /identity/v3/projects/2ef37f3c7d514ccc96ad78d4ed7c4f6c/users/3ba140fc89594c2b89be8d73f9b1d8f6/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:31.602059 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3371ba-d290-431b-aadf-64b23e538f91 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:31.606691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852442b6-a785-4b71-92d3-d5f284e92276 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:31.637782 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:31.671547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3371ba-d290-431b-aadf-64b23e538f91 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:31.678173 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82de1005-5247-4b41-8152-6085b8f63ff2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:31.678933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82de1005-5247-4b41-8152-6085b8f63ff2 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:31.679214 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82de1005-5247-4b41-8152-6085b8f63ff2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:31.679490 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82de1005-5247-4b41-8152-6085b8f63ff2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:31.680153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82de1005-5247-4b41-8152-6085b8f63ff2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:31.681779 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82de1005-5247-4b41-8152-6085b8f63ff2 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:31.686737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72c5843-7294-4aa7-9e9f-dc0384ccf7f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:31.687539 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72c5843-7294-4aa7-9e9f-dc0384ccf7f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:31.688020 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72c5843-7294-4aa7-9e9f-dc0384ccf7f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:31.688434 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72c5843-7294-4aa7-9e9f-dc0384ccf7f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:31.689220 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72c5843-7294-4aa7-9e9f-dc0384ccf7f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:31.691240 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72c5843-7294-4aa7-9e9f-dc0384ccf7f3 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:31.692751 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1065/2121] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:31 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:58:31.697028 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82de1005-5247-4b41-8152-6085b8f63ff2 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:31.707034 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1057/2122] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:31 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:58:31.708674 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72c5843-7294-4aa7-9e9f-dc0384ccf7f3 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:31.711157 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82de1005-5247-4b41-8152-6085b8f63ff2 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:31.715961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a08aad-96a7-4608-b004-d951efc02296 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:31.716657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a08aad-96a7-4608-b004-d951efc02296 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:31.716937 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a08aad-96a7-4608-b004-d951efc02296 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:31.717212 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a08aad-96a7-4608-b004-d951efc02296 admin admin] PATH_INFO: `/v3/projects/2ef37f3c7d514ccc96ad78d4ed7c4f6c/users/3ba140fc89594c2b89be8d73f9b1d8f6/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:31.719138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a08aad-96a7-4608-b004-d951efc02296 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2ef37f3c7d514ccc96ad78d4ed7c4f6c, user_id=3ba140fc89594c2b89be8d73f9b1d8f6, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:31.720817 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72c5843-7294-4aa7-9e9f-dc0384ccf7f3 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:31.721776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a08aad-96a7-4608-b004-d951efc02296 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:31.728692 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1728fbb3-37ce-453c-bd49-7c80e04c555f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:31.729381 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1728fbb3-37ce-453c-bd49-7c80e04c555f admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:31.729660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1728fbb3-37ce-453c-bd49-7c80e04c555f admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:31.729931 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1728fbb3-37ce-453c-bd49-7c80e04c555f admin admin] PATH_INFO: `/v3/projects/e94070a56c9c42c79ff6be6a261c6def/users/3ca7e55c90cd4a71b03804658cc43922/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:31.737583 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1066/2123] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:31 2022] PUT /identity/v3/projects/2ef37f3c7d514ccc96ad78d4ed7c4f6c/users/3ba140fc89594c2b89be8d73f9b1d8f6/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:31.749110 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c089e8b6-1387-4210-9119-17920cad18fd None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:31.749312 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c089e8b6-1387-4210-9119-17920cad18fd None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:31.749586 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c089e8b6-1387-4210-9119-17920cad18fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:31.751649 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1728fbb3-37ce-453c-bd49-7c80e04c555f admin admin] RBAC: Authorizing `identity:create_grant(project_id=e94070a56c9c42c79ff6be6a261c6def, user_id=3ca7e55c90cd4a71b03804658cc43922, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:31.754367 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1728fbb3-37ce-453c-bd49-7c80e04c555f admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:31.758150 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:31.772163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1058/2124] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:31 2022] PUT /identity/v3/projects/e94070a56c9c42c79ff6be6a261c6def/users/3ca7e55c90cd4a71b03804658cc43922/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:31.782399 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9846117a-5e3b-4373-8525-40ff78a463a2 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:31.782718 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9846117a-5e3b-4373-8525-40ff78a463a2 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:31.783114 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9846117a-5e3b-4373-8525-40ff78a463a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:31.785896 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-c089e8b6-1387-4210-9119-17920cad18fd None None] MFA Rules not processed for user `3ba140fc89594c2b89be8d73f9b1d8f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:31.787762 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c089e8b6-1387-4210-9119-17920cad18fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:31.820125 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-9846117a-5e3b-4373-8525-40ff78a463a2 None None] MFA Rules not processed for user `3ca7e55c90cd4a71b03804658cc43922`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:31.824146 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-9846117a-5e3b-4373-8525-40ff78a463a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:31.885505 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1067/2125] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:31 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:58:31.896631 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c089e8b6-1387-4210-9119-17920cad18fd None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:31.898350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c089e8b6-1387-4210-9119-17920cad18fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:31.913290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1059/2126] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:31 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:58:31.917735 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9846117a-5e3b-4373-8525-40ff78a463a2 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:31.963948 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c089e8b6-1387-4210-9119-17920cad18fd None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:31.968170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a76e52-fadd-44e1-8c85-5c1c90f359f6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:31.968836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a76e52-fadd-44e1-8c85-5c1c90f359f6 service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:31.969315 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a76e52-fadd-44e1-8c85-5c1c90f359f6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:31.969703 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a76e52-fadd-44e1-8c85-5c1c90f359f6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:31.997500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9846117a-5e3b-4373-8525-40ff78a463a2 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:32.002085 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeab7a72-5465-44cc-afd2-a71e47587f69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:32.002730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeab7a72-5465-44cc-afd2-a71e47587f69 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:32.003011 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeab7a72-5465-44cc-afd2-a71e47587f69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:32.003283 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeab7a72-5465-44cc-afd2-a71e47587f69 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:32.003916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeab7a72-5465-44cc-afd2-a71e47587f69 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:32.006415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeab7a72-5465-44cc-afd2-a71e47587f69 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:32.036388 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1060/2127] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:31 2022] POST /identity/v3/projects => generated 377 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:58:32.036593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:32.039368 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a76e52-fadd-44e1-8c85-5c1c90f359f6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:32.040605 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeab7a72-5465-44cc-afd2-a71e47587f69 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:32.040935 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a76e52-fadd-44e1-8c85-5c1c90f359f6 service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:32.051698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1068/2128] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:31 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:58:32.122562 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeab7a72-5465-44cc-afd2-a71e47587f69 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:32.127770 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6bcde9-904c-466c-8701-0484b26673b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:32.128407 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6bcde9-904c-466c-8701-0484b26673b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:32.128713 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6bcde9-904c-466c-8701-0484b26673b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:32.128951 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6bcde9-904c-466c-8701-0484b26673b9 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:32.129599 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6bcde9-904c-466c-8701-0484b26673b9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:32.134084 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6bcde9-904c-466c-8701-0484b26673b9 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:32.149962 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:32.155435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1061/2129] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:32 2022] POST /identity/v3/users => generated 289 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:32.159431 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a76e52-fadd-44e1-8c85-5c1c90f359f6 service cinder] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:32.171024 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a76e52-fadd-44e1-8c85-5c1c90f359f6 service cinder] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:32.175998 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce068958-b572-4dff-ad14-9567e7758d72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:32.176696 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce068958-b572-4dff-ad14-9567e7758d72 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:32.177069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce068958-b572-4dff-ad14-9567e7758d72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:32.177347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce068958-b572-4dff-ad14-9567e7758d72 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:32.178299 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce068958-b572-4dff-ad14-9567e7758d72 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:32.179634 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce068958-b572-4dff-ad14-9567e7758d72 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:32.190663 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1069/2130] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:32 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:32.195260 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6bcde9-904c-466c-8701-0484b26673b9 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:32.205725 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6bcde9-904c-466c-8701-0484b26673b9 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:32.211112 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1280b6-793c-4c94-8e1c-9fbf89c8ccbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:32.211713 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1280b6-793c-4c94-8e1c-9fbf89c8ccbe admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:32.211988 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1280b6-793c-4c94-8e1c-9fbf89c8ccbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:32.212257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1280b6-793c-4c94-8e1c-9fbf89c8ccbe admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:32.212989 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1280b6-793c-4c94-8e1c-9fbf89c8ccbe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:32.214584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1280b6-793c-4c94-8e1c-9fbf89c8ccbe admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:32.228505 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1062/2131] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:32 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:32.230768 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce068958-b572-4dff-ad14-9567e7758d72 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:32.243702 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce068958-b572-4dff-ad14-9567e7758d72 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:32.249898 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f523c43c-33d3-4b29-863d-4509d6f9c012 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:32.250618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f523c43c-33d3-4b29-863d-4509d6f9c012 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:32.250900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f523c43c-33d3-4b29-863d-4509d6f9c012 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:32.251184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f523c43c-33d3-4b29-863d-4509d6f9c012 admin admin] PATH_INFO: `/v3/projects/3c56265c2d064e5cb8036fe9c6214070/users/a718963c71624eb38e5d372ca55591ca/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:32.269044 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f523c43c-33d3-4b29-863d-4509d6f9c012 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3c56265c2d064e5cb8036fe9c6214070, user_id=a718963c71624eb38e5d372ca55591ca, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:32.273193 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f523c43c-33d3-4b29-863d-4509d6f9c012 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:32.291022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1070/2132] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:32 2022] PUT /identity/v3/projects/3c56265c2d064e5cb8036fe9c6214070/users/a718963c71624eb38e5d372ca55591ca/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:32.302258 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d340252d-efc0-485e-a4fd-5e9360328741 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:32.302668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d340252d-efc0-485e-a4fd-5e9360328741 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:32.302938 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d340252d-efc0-485e-a4fd-5e9360328741 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:32.343390 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-d340252d-efc0-485e-a4fd-5e9360328741 None None] MFA Rules not processed for user `a718963c71624eb38e5d372ca55591ca`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:32.344691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d340252d-efc0-485e-a4fd-5e9360328741 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:32.443501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1063/2133] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:32 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:32.448501 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f523c43c-33d3-4b29-863d-4509d6f9c012 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:32.515568 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f523c43c-33d3-4b29-863d-4509d6f9c012 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:32.520169 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52c4aaa-c11c-4365-8169-71e156e09ce1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:32.521278 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52c4aaa-c11c-4365-8169-71e156e09ce1 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:32.521551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52c4aaa-c11c-4365-8169-71e156e09ce1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:32.521820 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52c4aaa-c11c-4365-8169-71e156e09ce1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:32.522558 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52c4aaa-c11c-4365-8169-71e156e09ce1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:32.524473 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52c4aaa-c11c-4365-8169-71e156e09ce1 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:32.553707 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1071/2134] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:32 2022] POST /identity/v3/projects => generated 375 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:58:32.558198 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d340252d-efc0-485e-a4fd-5e9360328741 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:32.570191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d340252d-efc0-485e-a4fd-5e9360328741 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:32.626238 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edbc0ea-52bd-4ef3-a8db-8b8f7215e550 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:32.626875 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edbc0ea-52bd-4ef3-a8db-8b8f7215e550 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:32.627217 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edbc0ea-52bd-4ef3-a8db-8b8f7215e550 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:32.627488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edbc0ea-52bd-4ef3-a8db-8b8f7215e550 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:32.628113 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edbc0ea-52bd-4ef3-a8db-8b8f7215e550 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:32.630644 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edbc0ea-52bd-4ef3-a8db-8b8f7215e550 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:32.656823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1064/2135] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:32 2022] POST /identity/v3/users => generated 294 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:32.660872 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52c4aaa-c11c-4365-8169-71e156e09ce1 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:32.672060 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52c4aaa-c11c-4365-8169-71e156e09ce1 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:32.676902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408403a1-4f13-4734-9621-ee2a82a71680 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:32.677503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408403a1-4f13-4734-9621-ee2a82a71680 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:32.677779 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408403a1-4f13-4734-9621-ee2a82a71680 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:32.678047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408403a1-4f13-4734-9621-ee2a82a71680 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:32.678711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408403a1-4f13-4734-9621-ee2a82a71680 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:32.680345 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408403a1-4f13-4734-9621-ee2a82a71680 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:32.691808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1072/2136] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:32 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:32.696543 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edbc0ea-52bd-4ef3-a8db-8b8f7215e550 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:32.707015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edbc0ea-52bd-4ef3-a8db-8b8f7215e550 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:32.712181 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc0e926-5c32-4fa5-af3b-aaa029012dc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:32.712831 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc0e926-5c32-4fa5-af3b-aaa029012dc2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:32.713399 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc0e926-5c32-4fa5-af3b-aaa029012dc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:32.713719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc0e926-5c32-4fa5-af3b-aaa029012dc2 admin admin] PATH_INFO: `/v3/projects/fc49f24582c045a9ab96408677e82506/users/1f0a15f3365e402e8fa6ad4c296e58b2/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:32.728111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc0e926-5c32-4fa5-af3b-aaa029012dc2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc49f24582c045a9ab96408677e82506, user_id=1f0a15f3365e402e8fa6ad4c296e58b2, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:32.730787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc0e926-5c32-4fa5-af3b-aaa029012dc2 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:32.743976 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1065/2137] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:32 2022] PUT /identity/v3/projects/fc49f24582c045a9ab96408677e82506/users/1f0a15f3365e402e8fa6ad4c296e58b2/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:32.747932 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408403a1-4f13-4734-9621-ee2a82a71680 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:32.760207 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408403a1-4f13-4734-9621-ee2a82a71680 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:32.816076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f919fe-67be-4884-9d7f-5399f04d4a7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:32.816683 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f919fe-67be-4884-9d7f-5399f04d4a7b admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:32.816963 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f919fe-67be-4884-9d7f-5399f04d4a7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:32.817236 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f919fe-67be-4884-9d7f-5399f04d4a7b admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:32.818412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f919fe-67be-4884-9d7f-5399f04d4a7b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:32.820084 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f919fe-67be-4884-9d7f-5399f04d4a7b admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:32.829968 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1073/2138] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:32 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:32.834602 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc0e926-5c32-4fa5-af3b-aaa029012dc2 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:32.844959 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc0e926-5c32-4fa5-af3b-aaa029012dc2 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:32.849376 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217fcb29-2a86-402b-ba7c-cc8f88298e44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:32.850159 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217fcb29-2a86-402b-ba7c-cc8f88298e44 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:32.850531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217fcb29-2a86-402b-ba7c-cc8f88298e44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:32.850834 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217fcb29-2a86-402b-ba7c-cc8f88298e44 admin admin] PATH_INFO: `/v3/projects/fc49f24582c045a9ab96408677e82506/users/1f0a15f3365e402e8fa6ad4c296e58b2/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:32.852827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217fcb29-2a86-402b-ba7c-cc8f88298e44 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc49f24582c045a9ab96408677e82506, user_id=1f0a15f3365e402e8fa6ad4c296e58b2, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:32.855580 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217fcb29-2a86-402b-ba7c-cc8f88298e44 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:32.867508 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1066/2139] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:32 2022] PUT /identity/v3/projects/fc49f24582c045a9ab96408677e82506/users/1f0a15f3365e402e8fa6ad4c296e58b2/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:32.877593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56baf2fa-9c0d-4280-81c1-95086ce47363 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:32.877879 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56baf2fa-9c0d-4280-81c1-95086ce47363 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:32.878200 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56baf2fa-9c0d-4280-81c1-95086ce47363 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:32.907631 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-56baf2fa-9c0d-4280-81c1-95086ce47363 None None] MFA Rules not processed for user `1f0a15f3365e402e8fa6ad4c296e58b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:32.908925 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-56baf2fa-9c0d-4280-81c1-95086ce47363 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:32.930371 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:32.991805 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1074/2140] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:32 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:33.004091 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217fcb29-2a86-402b-ba7c-cc8f88298e44 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:33.006813 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-217fcb29-2a86-402b-ba7c-cc8f88298e44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:33.058883 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:33.085616 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217fcb29-2a86-402b-ba7c-cc8f88298e44 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:33.089604 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77357ee-c113-4796-acde-7fe0fe9f4d92 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:33.090214 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77357ee-c113-4796-acde-7fe0fe9f4d92 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:33.090514 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77357ee-c113-4796-acde-7fe0fe9f4d92 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:33.090783 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77357ee-c113-4796-acde-7fe0fe9f4d92 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:33.150071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77357ee-c113-4796-acde-7fe0fe9f4d92 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:33.151699 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77357ee-c113-4796-acde-7fe0fe9f4d92 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:33.160968 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1067/2141] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:33 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:34.732695 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56baf2fa-9c0d-4280-81c1-95086ce47363 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:34.746682 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56baf2fa-9c0d-4280-81c1-95086ce47363 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:34.757914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33308d51-f804-4ef1-8f22-cee516ae051f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:34.759820 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33308d51-f804-4ef1-8f22-cee516ae051f service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:34.760918 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33308d51-f804-4ef1-8f22-cee516ae051f service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:34.760918 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33308d51-f804-4ef1-8f22-cee516ae051f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:34.836591 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33308d51-f804-4ef1-8f22-cee516ae051f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:34.838031 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33308d51-f804-4ef1-8f22-cee516ae051f service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:34.847143 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1075/2142] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:34 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:35.479745 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77357ee-c113-4796-acde-7fe0fe9f4d92 service cinder] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:35.489144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77357ee-c113-4796-acde-7fe0fe9f4d92 service cinder] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:35.493295 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5440105c-6522-4ed0-8491-52b9673c320d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:35.493941 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5440105c-6522-4ed0-8491-52b9673c320d service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:35.494251 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5440105c-6522-4ed0-8491-52b9673c320d service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:35.494579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5440105c-6522-4ed0-8491-52b9673c320d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:35.505477 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5440105c-6522-4ed0-8491-52b9673c320d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:35.506852 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5440105c-6522-4ed0-8491-52b9673c320d service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:35.517234 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1068/2143] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:35 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:38.923593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33308d51-f804-4ef1-8f22-cee516ae051f service cinder] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:38.939292 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33308d51-f804-4ef1-8f22-cee516ae051f service cinder] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:38.945344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff6de50-6c42-41db-8028-5287727877bd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:38.946388 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff6de50-6c42-41db-8028-5287727877bd service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:38.946922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff6de50-6c42-41db-8028-5287727877bd service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:38.947429 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff6de50-6c42-41db-8028-5287727877bd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:39.051607 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff6de50-6c42-41db-8028-5287727877bd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:39.053958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff6de50-6c42-41db-8028-5287727877bd service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:39.064654 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1076/2144] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:38 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:45.031439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5440105c-6522-4ed0-8491-52b9673c320d service cinder] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:45.033090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-5440105c-6522-4ed0-8491-52b9673c320d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:45.104957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5440105c-6522-4ed0-8491-52b9673c320d service cinder] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:45.109484 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0905637-64c0-41f6-8ae4-43734ecf70f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:45.110253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0905637-64c0-41f6-8ae4-43734ecf70f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:45.110560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0905637-64c0-41f6-8ae4-43734ecf70f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:45.110953 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0905637-64c0-41f6-8ae4-43734ecf70f1 admin admin] PATH_INFO: `/v3/users/5bba915f4a374bba9c8f934b3f742b6c` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:45.112078 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0905637-64c0-41f6-8ae4-43734ecf70f1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5bba915f4a374bba9c8f934b3f742b6c)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:45.113947 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0905637-64c0-41f6-8ae4-43734ecf70f1 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:45.168916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-a0905637-64c0-41f6-8ae4-43734ecf70f1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5bba915f4a374bba9c8f934b3f742b6c'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:45.177621 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-a0905637-64c0-41f6-8ae4-43734ecf70f1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5bba915f4a374bba9c8f934b3f742b6c'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:45.178352 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-a0905637-64c0-41f6-8ae4-43734ecf70f1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5bba915f4a374bba9c8f934b3f742b6c'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:45.178993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-a0905637-64c0-41f6-8ae4-43734ecf70f1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5bba915f4a374bba9c8f934b3f742b6c'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:45.186324 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-a0905637-64c0-41f6-8ae4-43734ecf70f1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5bba915f4a374bba9c8f934b3f742b6c'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:45.195962 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:45.209969 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1069/2145] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:45 2022] DELETE /identity/v3/users/5bba915f4a374bba9c8f934b3f742b6c => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:45.215179 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d769fc9-df1f-4e9f-91b6-751c0e7a5937 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:45.215682 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d769fc9-df1f-4e9f-91b6-751c0e7a5937 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:45.215778 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d769fc9-df1f-4e9f-91b6-751c0e7a5937 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:45.235541 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-4d769fc9-df1f-4e9f-91b6-751c0e7a5937 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:45.236753 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-4d769fc9-df1f-4e9f-91b6-751c0e7a5937 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:45.296328 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:45.316533 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1077/2146] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:58:45.327720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0905637-64c0-41f6-8ae4-43734ecf70f1 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:45.329198 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-a0905637-64c0-41f6-8ae4-43734ecf70f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:45.397132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0905637-64c0-41f6-8ae4-43734ecf70f1 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:45.400796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096946c4-d938-4afd-8c3c-ff4d9dff8d30 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:45.401406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096946c4-d938-4afd-8c3c-ff4d9dff8d30 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:45.401687 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096946c4-d938-4afd-8c3c-ff4d9dff8d30 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:45.401960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096946c4-d938-4afd-8c3c-ff4d9dff8d30 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:45.412231 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096946c4-d938-4afd-8c3c-ff4d9dff8d30 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:45.413665 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096946c4-d938-4afd-8c3c-ff4d9dff8d30 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:45.423667 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1070/2147] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:58:45.701563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d769fc9-df1f-4e9f-91b6-751c0e7a5937 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:45.703035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-4d769fc9-df1f-4e9f-91b6-751c0e7a5937 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:45.721630 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d769fc9-df1f-4e9f-91b6-751c0e7a5937 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:45.727449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4445124d-e962-4151-9dad-4172a3e92291 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:45.728766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4445124d-e962-4151-9dad-4172a3e92291 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:45.729188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4445124d-e962-4151-9dad-4172a3e92291 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:45.730564 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4445124d-e962-4151-9dad-4172a3e92291 admin admin] PATH_INFO: `/v3/projects/c107c89986ca4ec7adb56f731a567459` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:45.731927 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4445124d-e962-4151-9dad-4172a3e92291 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c107c89986ca4ec7adb56f731a567459)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:45.734027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4445124d-e962-4151-9dad-4172a3e92291 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:45.779040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096946c4-d938-4afd-8c3c-ff4d9dff8d30 service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:45.780371 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-096946c4-d938-4afd-8c3c-ff4d9dff8d30 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:45.787536 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-4445124d-e962-4151-9dad-4172a3e92291 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c107c89986ca4ec7adb56f731a567459', 'request_id': 'req-4445124d-e962-4151-9dad-4172a3e92291'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:45.793657 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-4445124d-e962-4151-9dad-4172a3e92291 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c107c89986ca4ec7adb56f731a567459', 'request_id': 'req-4445124d-e962-4151-9dad-4172a3e92291'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:45.794809 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096946c4-d938-4afd-8c3c-ff4d9dff8d30 service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:45.803567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-4445124d-e962-4151-9dad-4172a3e92291 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c107c89986ca4ec7adb56f731a567459', 'request_id': 'req-4445124d-e962-4151-9dad-4172a3e92291'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:45.815273 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-4445124d-e962-4151-9dad-4172a3e92291 admin admin] The token cache is being invalidate because project c107c89986ca4ec7adb56f731a567459 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:45.815599 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-4445124d-e962-4151-9dad-4172a3e92291 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:45.816301 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-4445124d-e962-4151-9dad-4172a3e92291 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:45.817890 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1078/2148] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:45 2022] DELETE /identity/v3/projects/c107c89986ca4ec7adb56f731a567459 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:45.822885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bc86d7-b471-4804-a172-0ab51c53a7ac None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:45.823114 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bc86d7-b471-4804-a172-0ab51c53a7ac None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:45.823651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bc86d7-b471-4804-a172-0ab51c53a7ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:45.843322 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-28bc86d7-b471-4804-a172-0ab51c53a7ac None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:45.844544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-28bc86d7-b471-4804-a172-0ab51c53a7ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:45.855539 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6896f09-502f-4969-b2ad-6dc64db3a0e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:45.856180 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6896f09-502f-4969-b2ad-6dc64db3a0e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:45.856468 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6896f09-502f-4969-b2ad-6dc64db3a0e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:45.856750 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6896f09-502f-4969-b2ad-6dc64db3a0e7 admin admin] PATH_INFO: `/v3/users/3ca7e55c90cd4a71b03804658cc43922` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:45.857889 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6896f09-502f-4969-b2ad-6dc64db3a0e7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3ca7e55c90cd4a71b03804658cc43922)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:45.859913 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6896f09-502f-4969-b2ad-6dc64db3a0e7 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:45.907494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b6896f09-502f-4969-b2ad-6dc64db3a0e7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3ca7e55c90cd4a71b03804658cc43922'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:45.907771 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1079/2149] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:58:45.915291 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b6896f09-502f-4969-b2ad-6dc64db3a0e7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3ca7e55c90cd4a71b03804658cc43922'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:45.916209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b6896f09-502f-4969-b2ad-6dc64db3a0e7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3ca7e55c90cd4a71b03804658cc43922'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:45.916372 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f925b6-e941-4ebf-96ea-e1024ece87d9 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:45.916606 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f925b6-e941-4ebf-96ea-e1024ece87d9 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:45.916904 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f925b6-e941-4ebf-96ea-e1024ece87d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:45.917003 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b6896f09-502f-4969-b2ad-6dc64db3a0e7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3ca7e55c90cd4a71b03804658cc43922'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:45.924611 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b6896f09-502f-4969-b2ad-6dc64db3a0e7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3ca7e55c90cd4a71b03804658cc43922'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:45.936424 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-07f925b6-e941-4ebf-96ea-e1024ece87d9 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:45.937618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-07f925b6-e941-4ebf-96ea-e1024ece87d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:45.944386 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1071/2150] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:45 2022] DELETE /identity/v3/users/3ca7e55c90cd4a71b03804658cc43922 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:45.946321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1080/2151] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:45.950080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28730838-0695-44b2-9cf7-843048605304 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:45.950080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28730838-0695-44b2-9cf7-843048605304 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:45.950080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28730838-0695-44b2-9cf7-843048605304 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:45.951351 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f925b6-e941-4ebf-96ea-e1024ece87d9 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:45.961538 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f925b6-e941-4ebf-96ea-e1024ece87d9 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:45.969441 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-28730838-0695-44b2-9cf7-843048605304 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:45.970932 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-28730838-0695-44b2-9cf7-843048605304 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:45.981416 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:46.030638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7b213d-57fd-40f2-ae0b-12da2512a3c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:46.031429 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7b213d-57fd-40f2-ae0b-12da2512a3c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:46.031980 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7b213d-57fd-40f2-ae0b-12da2512a3c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:46.032258 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7b213d-57fd-40f2-ae0b-12da2512a3c1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:46.032988 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7b213d-57fd-40f2-ae0b-12da2512a3c1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:58:46.033299 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7b213d-57fd-40f2-ae0b-12da2512a3c1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:46.035019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7b213d-57fd-40f2-ae0b-12da2512a3c1 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:46.048753 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:46.048753 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1081/2152] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:58:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:58:46.053120 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1072/2153] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:46.054073 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7b213d-57fd-40f2-ae0b-12da2512a3c1 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:46.063703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28730838-0695-44b2-9cf7-843048605304 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:46.065111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-28730838-0695-44b2-9cf7-843048605304 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:46.066410 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7b213d-57fd-40f2-ae0b-12da2512a3c1 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:46.073096 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0716df50-193e-4324-8423-5d32dc552991 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:46.074111 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0716df50-193e-4324-8423-5d32dc552991 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:46.074394 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0716df50-193e-4324-8423-5d32dc552991 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:46.074710 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0716df50-193e-4324-8423-5d32dc552991 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:46.075350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0716df50-193e-4324-8423-5d32dc552991 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:46.077267 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0716df50-193e-4324-8423-5d32dc552991 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:46.112900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1082/2154] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:46 2022] POST /identity/v3/projects => generated 365 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:46.117311 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0716df50-193e-4324-8423-5d32dc552991 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:46.133840 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28730838-0695-44b2-9cf7-843048605304 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:46.184358 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88f79c9-ec73-4f09-8463-bd34f6654ffe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:46.184966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88f79c9-ec73-4f09-8463-bd34f6654ffe service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:46.185243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88f79c9-ec73-4f09-8463-bd34f6654ffe service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:46.185542 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88f79c9-ec73-4f09-8463-bd34f6654ffe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:46.186927 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0716df50-193e-4324-8423-5d32dc552991 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:46.191578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423e0c07-de7a-4c97-967a-a7ecf6cdf3ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:46.192176 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423e0c07-de7a-4c97-967a-a7ecf6cdf3ab admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:46.192575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423e0c07-de7a-4c97-967a-a7ecf6cdf3ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:46.192846 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423e0c07-de7a-4c97-967a-a7ecf6cdf3ab admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:46.193507 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423e0c07-de7a-4c97-967a-a7ecf6cdf3ab admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:46.195556 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423e0c07-de7a-4c97-967a-a7ecf6cdf3ab admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:46.202879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88f79c9-ec73-4f09-8463-bd34f6654ffe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:46.204358 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88f79c9-ec73-4f09-8463-bd34f6654ffe service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:46.215350 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1073/2155] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:46.222494 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1083/2156] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:46 2022] POST /identity/v3/users => generated 283 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:46.226116 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88f79c9-ec73-4f09-8463-bd34f6654ffe service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:46.236357 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88f79c9-ec73-4f09-8463-bd34f6654ffe service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:46.241337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e09677-3786-45ea-b01b-d6ad0d22a30c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:46.241948 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e09677-3786-45ea-b01b-d6ad0d22a30c admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:46.242230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e09677-3786-45ea-b01b-d6ad0d22a30c admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:46.243159 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e09677-3786-45ea-b01b-d6ad0d22a30c admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:46.243819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e09677-3786-45ea-b01b-d6ad0d22a30c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:46.245548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e09677-3786-45ea-b01b-d6ad0d22a30c admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:46.256895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1074/2157] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:46 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:46.260883 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423e0c07-de7a-4c97-967a-a7ecf6cdf3ab admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:46.272435 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423e0c07-de7a-4c97-967a-a7ecf6cdf3ab admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:46.278236 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66dae048-9104-4956-80f6-a7f4dd4bf6ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:46.278895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66dae048-9104-4956-80f6-a7f4dd4bf6ac admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:46.279176 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66dae048-9104-4956-80f6-a7f4dd4bf6ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:46.279448 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66dae048-9104-4956-80f6-a7f4dd4bf6ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:46.280102 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66dae048-9104-4956-80f6-a7f4dd4bf6ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:46.281735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66dae048-9104-4956-80f6-a7f4dd4bf6ac admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:46.293849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1084/2158] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:46 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:46.298582 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e09677-3786-45ea-b01b-d6ad0d22a30c admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:46.308677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e09677-3786-45ea-b01b-d6ad0d22a30c admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:46.313348 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7dc2d8-fc7f-4015-94c4-aa2a6eab486f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:46.313999 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7dc2d8-fc7f-4015-94c4-aa2a6eab486f admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:46.314283 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7dc2d8-fc7f-4015-94c4-aa2a6eab486f admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:46.314598 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7dc2d8-fc7f-4015-94c4-aa2a6eab486f admin admin] PATH_INFO: `/v3/projects/efccc3a3c8254b15bf40f003f37832af/users/c33fc09990e0453e8c17c80ad660bc32/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:46.331442 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7dc2d8-fc7f-4015-94c4-aa2a6eab486f admin admin] RBAC: Authorizing `identity:create_grant(project_id=efccc3a3c8254b15bf40f003f37832af, user_id=c33fc09990e0453e8c17c80ad660bc32, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:46.334799 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7dc2d8-fc7f-4015-94c4-aa2a6eab486f admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:46.349151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1075/2159] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:46 2022] PUT /identity/v3/projects/efccc3a3c8254b15bf40f003f37832af/users/c33fc09990e0453e8c17c80ad660bc32/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:46.359696 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7969be3-18bb-4309-86fa-775e5488a825 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:46.359972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7969be3-18bb-4309-86fa-775e5488a825 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:46.360240 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7969be3-18bb-4309-86fa-775e5488a825 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:46.392083 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-e7969be3-18bb-4309-86fa-775e5488a825 None None] MFA Rules not processed for user `c33fc09990e0453e8c17c80ad660bc32`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:46.393412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-e7969be3-18bb-4309-86fa-775e5488a825 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:46.467915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1085/2160] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:46 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:46.479747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7dc2d8-fc7f-4015-94c4-aa2a6eab486f admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:46.481486 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8f7dc2d8-fc7f-4015-94c4-aa2a6eab486f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:46.500835 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7969be3-18bb-4309-86fa-775e5488a825 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:46.503037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-e7969be3-18bb-4309-86fa-775e5488a825 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:46.543134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:46.564666 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7dc2d8-fc7f-4015-94c4-aa2a6eab486f admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:46.571785 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa9c6de-9495-463c-a1a3-384c50dd6136 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:46.572425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa9c6de-9495-463c-a1a3-384c50dd6136 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:46.572702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa9c6de-9495-463c-a1a3-384c50dd6136 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:46.572976 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa9c6de-9495-463c-a1a3-384c50dd6136 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:46.574329 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:46.583366 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa9c6de-9495-463c-a1a3-384c50dd6136 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:46.584837 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa9c6de-9495-463c-a1a3-384c50dd6136 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:46.593616 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7969be3-18bb-4309-86fa-775e5488a825 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:46.595016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1076/2161] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:46 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:46.598752 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59d450a-c8bf-4cbd-b845-1dd3547be70f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:46.599401 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59d450a-c8bf-4cbd-b845-1dd3547be70f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:46.599685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59d450a-c8bf-4cbd-b845-1dd3547be70f admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:46.599960 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59d450a-c8bf-4cbd-b845-1dd3547be70f admin admin] PATH_INFO: `/v3/projects/e94070a56c9c42c79ff6be6a261c6def` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:46.601206 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59d450a-c8bf-4cbd-b845-1dd3547be70f admin admin] RBAC: Authorizing `identity:delete_project(project_id=e94070a56c9c42c79ff6be6a261c6def)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:46.603185 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59d450a-c8bf-4cbd-b845-1dd3547be70f admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:46.655827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d59d450a-c8bf-4cbd-b845-1dd3547be70f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e94070a56c9c42c79ff6be6a261c6def', 'request_id': 'req-d59d450a-c8bf-4cbd-b845-1dd3547be70f'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:46.661912 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d59d450a-c8bf-4cbd-b845-1dd3547be70f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e94070a56c9c42c79ff6be6a261c6def', 'request_id': 'req-d59d450a-c8bf-4cbd-b845-1dd3547be70f'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:46.671796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d59d450a-c8bf-4cbd-b845-1dd3547be70f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e94070a56c9c42c79ff6be6a261c6def', 'request_id': 'req-d59d450a-c8bf-4cbd-b845-1dd3547be70f'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:46.684897 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d59d450a-c8bf-4cbd-b845-1dd3547be70f admin admin] The token cache is being invalidate because project e94070a56c9c42c79ff6be6a261c6def was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:46.684897 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d59d450a-c8bf-4cbd-b845-1dd3547be70f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:46.685581 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d59d450a-c8bf-4cbd-b845-1dd3547be70f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:46.687134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1086/2162] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:46 2022] DELETE /identity/v3/projects/e94070a56c9c42c79ff6be6a261c6def => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:58:46.691338 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa9c6de-9495-463c-a1a3-384c50dd6136 service cinder] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:46.703339 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa9c6de-9495-463c-a1a3-384c50dd6136 service cinder] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:46.765984 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a01b347-a53e-48f7-8b50-d2fc5e4b7e35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:46.766763 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a01b347-a53e-48f7-8b50-d2fc5e4b7e35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:46.767042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a01b347-a53e-48f7-8b50-d2fc5e4b7e35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:46.767739 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a01b347-a53e-48f7-8b50-d2fc5e4b7e35 admin admin] PATH_INFO: `/v3/users/a718963c71624eb38e5d372ca55591ca` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:46.768921 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a01b347-a53e-48f7-8b50-d2fc5e4b7e35 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a718963c71624eb38e5d372ca55591ca)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:46.771318 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a01b347-a53e-48f7-8b50-d2fc5e4b7e35 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:46.823511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8a01b347-a53e-48f7-8b50-d2fc5e4b7e35 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a718963c71624eb38e5d372ca55591ca'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:46.831683 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8a01b347-a53e-48f7-8b50-d2fc5e4b7e35 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a718963c71624eb38e5d372ca55591ca'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:46.832342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8a01b347-a53e-48f7-8b50-d2fc5e4b7e35 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a718963c71624eb38e5d372ca55591ca'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:46.833056 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8a01b347-a53e-48f7-8b50-d2fc5e4b7e35 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a718963c71624eb38e5d372ca55591ca'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:46.840489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8a01b347-a53e-48f7-8b50-d2fc5e4b7e35 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a718963c71624eb38e5d372ca55591ca'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:46.860208 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1077/2163] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:46 2022] DELETE /identity/v3/users/a718963c71624eb38e5d372ca55591ca => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:47.183759 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59d450a-c8bf-4cbd-b845-1dd3547be70f admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:47.185410 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-d59d450a-c8bf-4cbd-b845-1dd3547be70f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:47.209436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59d450a-c8bf-4cbd-b845-1dd3547be70f admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:47.277331 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de63d4e2-8a48-48c2-b16e-8699c515c3bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:47.278037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de63d4e2-8a48-48c2-b16e-8699c515c3bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:47.278382 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de63d4e2-8a48-48c2-b16e-8699c515c3bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:47.278691 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de63d4e2-8a48-48c2-b16e-8699c515c3bf admin admin] PATH_INFO: `/v3/projects/3c56265c2d064e5cb8036fe9c6214070` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:47.279826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de63d4e2-8a48-48c2-b16e-8699c515c3bf admin admin] RBAC: Authorizing `identity:delete_project(project_id=3c56265c2d064e5cb8036fe9c6214070)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:47.281724 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de63d4e2-8a48-48c2-b16e-8699c515c3bf admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:47.330210 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-de63d4e2-8a48-48c2-b16e-8699c515c3bf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3c56265c2d064e5cb8036fe9c6214070', 'request_id': 'req-de63d4e2-8a48-48c2-b16e-8699c515c3bf'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:47.336981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-de63d4e2-8a48-48c2-b16e-8699c515c3bf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3c56265c2d064e5cb8036fe9c6214070', 'request_id': 'req-de63d4e2-8a48-48c2-b16e-8699c515c3bf'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:47.345157 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-de63d4e2-8a48-48c2-b16e-8699c515c3bf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3c56265c2d064e5cb8036fe9c6214070', 'request_id': 'req-de63d4e2-8a48-48c2-b16e-8699c515c3bf'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:47.357000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-de63d4e2-8a48-48c2-b16e-8699c515c3bf admin admin] The token cache is being invalidate because project 3c56265c2d064e5cb8036fe9c6214070 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:47.357222 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-de63d4e2-8a48-48c2-b16e-8699c515c3bf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:47.357844 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-de63d4e2-8a48-48c2-b16e-8699c515c3bf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:47.359333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1087/2164] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:47 2022] DELETE /identity/v3/projects/3c56265c2d064e5cb8036fe9c6214070 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:47.363297 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a01b347-a53e-48f7-8b50-d2fc5e4b7e35 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:47.364828 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8a01b347-a53e-48f7-8b50-d2fc5e4b7e35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:47.380115 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a01b347-a53e-48f7-8b50-d2fc5e4b7e35 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:47.438391 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b39ed97-6d87-427a-8e08-3140836ead70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:47.439178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b39ed97-6d87-427a-8e08-3140836ead70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:47.439462 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b39ed97-6d87-427a-8e08-3140836ead70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:47.439730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b39ed97-6d87-427a-8e08-3140836ead70 admin admin] PATH_INFO: `/v3/users/1f0a15f3365e402e8fa6ad4c296e58b2` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:47.441003 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b39ed97-6d87-427a-8e08-3140836ead70 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1f0a15f3365e402e8fa6ad4c296e58b2)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:47.442982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b39ed97-6d87-427a-8e08-3140836ead70 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:47.444629 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:47.499996 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-2b39ed97-6d87-427a-8e08-3140836ead70 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1f0a15f3365e402e8fa6ad4c296e58b2'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:47.515453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-2b39ed97-6d87-427a-8e08-3140836ead70 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1f0a15f3365e402e8fa6ad4c296e58b2'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:47.516449 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-2b39ed97-6d87-427a-8e08-3140836ead70 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1f0a15f3365e402e8fa6ad4c296e58b2'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:47.517260 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-2b39ed97-6d87-427a-8e08-3140836ead70 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1f0a15f3365e402e8fa6ad4c296e58b2'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:47.527267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-2b39ed97-6d87-427a-8e08-3140836ead70 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1f0a15f3365e402e8fa6ad4c296e58b2'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:47.545957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1078/2165] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:47 2022] DELETE /identity/v3/users/1f0a15f3365e402e8fa6ad4c296e58b2 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:47.944759 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de63d4e2-8a48-48c2-b16e-8699c515c3bf admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:47.946364 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-de63d4e2-8a48-48c2-b16e-8699c515c3bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:47.971932 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de63d4e2-8a48-48c2-b16e-8699c515c3bf admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:47.976154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:48.043473 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa32e7f9-f5d4-469c-9c56-bdc27659ed0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:48.044113 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa32e7f9-f5d4-469c-9c56-bdc27659ed0c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:48.044397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa32e7f9-f5d4-469c-9c56-bdc27659ed0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:48.044674 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa32e7f9-f5d4-469c-9c56-bdc27659ed0c admin admin] PATH_INFO: `/v3/projects/fc49f24582c045a9ab96408677e82506` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:48.045813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa32e7f9-f5d4-469c-9c56-bdc27659ed0c admin admin] RBAC: Authorizing `identity:delete_project(project_id=fc49f24582c045a9ab96408677e82506)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:48.047785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa32e7f9-f5d4-469c-9c56-bdc27659ed0c admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:48.099838 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-fa32e7f9-f5d4-469c-9c56-bdc27659ed0c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fc49f24582c045a9ab96408677e82506', 'request_id': 'req-fa32e7f9-f5d4-469c-9c56-bdc27659ed0c'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:48.105936 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-fa32e7f9-f5d4-469c-9c56-bdc27659ed0c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fc49f24582c045a9ab96408677e82506', 'request_id': 'req-fa32e7f9-f5d4-469c-9c56-bdc27659ed0c'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:48.113651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-fa32e7f9-f5d4-469c-9c56-bdc27659ed0c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fc49f24582c045a9ab96408677e82506', 'request_id': 'req-fa32e7f9-f5d4-469c-9c56-bdc27659ed0c'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:48.130631 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-fa32e7f9-f5d4-469c-9c56-bdc27659ed0c admin admin] The token cache is being invalidate because project fc49f24582c045a9ab96408677e82506 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:48.130988 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-fa32e7f9-f5d4-469c-9c56-bdc27659ed0c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:48.131647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-fa32e7f9-f5d4-469c-9c56-bdc27659ed0c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:48.133129 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1088/2166] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:47 2022] DELETE /identity/v3/projects/fc49f24582c045a9ab96408677e82506 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:48.138017 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9db5f4-1ef8-41ec-83e5-cb8d091992b0 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:48.138366 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9db5f4-1ef8-41ec-83e5-cb8d091992b0 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:48.138736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9db5f4-1ef8-41ec-83e5-cb8d091992b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:48.160205 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-3b9db5f4-1ef8-41ec-83e5-cb8d091992b0 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:48.162054 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-3b9db5f4-1ef8-41ec-83e5-cb8d091992b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:48.236840 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1079/2167] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:58:48.245444 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e42c451-c6ef-4c9e-9a10-9c60b0dc2c4c None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:48.245829 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e42c451-c6ef-4c9e-9a10-9c60b0dc2c4c None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:48.246287 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e42c451-c6ef-4c9e-9a10-9c60b0dc2c4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:48.276333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-0e42c451-c6ef-4c9e-9a10-9c60b0dc2c4c None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:48.278516 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-0e42c451-c6ef-4c9e-9a10-9c60b0dc2c4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:48.292234 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1089/2168] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:58:48.296460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9db5f4-1ef8-41ec-83e5-cb8d091992b0 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:48.307312 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9db5f4-1ef8-41ec-83e5-cb8d091992b0 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:48.311924 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53be1802-585c-4e21-95b0-3c462f065012 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:48.312620 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53be1802-585c-4e21-95b0-3c462f065012 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:48.312849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53be1802-585c-4e21-95b0-3c462f065012 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:48.313146 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53be1802-585c-4e21-95b0-3c462f065012 admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:48.313797 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53be1802-585c-4e21-95b0-3c462f065012 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:58:48.314101 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53be1802-585c-4e21-95b0-3c462f065012 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:48.315757 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53be1802-585c-4e21-95b0-3c462f065012 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:48.329189 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1080/2169] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:58:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:58:48.333842 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e42c451-c6ef-4c9e-9a10-9c60b0dc2c4c None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:48.343108 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e42c451-c6ef-4c9e-9a10-9c60b0dc2c4c None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:48.348037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55abb526-67a4-4f5b-807f-ec7795b245d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:48.349146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55abb526-67a4-4f5b-807f-ec7795b245d1 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:48.349421 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55abb526-67a4-4f5b-807f-ec7795b245d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:48.349694 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55abb526-67a4-4f5b-807f-ec7795b245d1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:48.350342 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55abb526-67a4-4f5b-807f-ec7795b245d1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:48.352304 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55abb526-67a4-4f5b-807f-ec7795b245d1 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:48.384760 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1090/2170] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:48 2022] POST /identity/v3/projects => generated 385 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:48.390313 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53be1802-585c-4e21-95b0-3c462f065012 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:48.403270 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53be1802-585c-4e21-95b0-3c462f065012 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:48.476160 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc039b4c-1635-4c89-8cd1-aae0c04fbd27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:48.476775 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc039b4c-1635-4c89-8cd1-aae0c04fbd27 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:48.477054 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc039b4c-1635-4c89-8cd1-aae0c04fbd27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:48.477323 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc039b4c-1635-4c89-8cd1-aae0c04fbd27 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:48.477968 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc039b4c-1635-4c89-8cd1-aae0c04fbd27 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:48.479972 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc039b4c-1635-4c89-8cd1-aae0c04fbd27 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:48.493201 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:48.509775 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55abb526-67a4-4f5b-807f-ec7795b245d1 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:48.518671 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1081/2171] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:48 2022] POST /identity/v3/users => generated 293 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:48.523200 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55abb526-67a4-4f5b-807f-ec7795b245d1 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:48.532333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77245817-ebb1-4822-a09a-bc5110b700d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:48.532977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77245817-ebb1-4822-a09a-bc5110b700d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:48.533256 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77245817-ebb1-4822-a09a-bc5110b700d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:48.533526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77245817-ebb1-4822-a09a-bc5110b700d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:48.534187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77245817-ebb1-4822-a09a-bc5110b700d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:48.535900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77245817-ebb1-4822-a09a-bc5110b700d8 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:48.547364 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1091/2172] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:48 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:48.551597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc039b4c-1635-4c89-8cd1-aae0c04fbd27 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:48.562946 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc039b4c-1635-4c89-8cd1-aae0c04fbd27 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:48.567654 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20360672-a5dc-4343-83c4-ee03e4992f6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:48.568250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20360672-a5dc-4343-83c4-ee03e4992f6d admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:48.568521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20360672-a5dc-4343-83c4-ee03e4992f6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:48.568795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20360672-a5dc-4343-83c4-ee03e4992f6d admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:48.569460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20360672-a5dc-4343-83c4-ee03e4992f6d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:48.571066 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20360672-a5dc-4343-83c4-ee03e4992f6d admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:48.585912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1082/2173] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:48 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:48.587873 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77245817-ebb1-4822-a09a-bc5110b700d8 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:48.598155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77245817-ebb1-4822-a09a-bc5110b700d8 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:48.603788 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10992f8-779b-46dd-9352-b62563ec16c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:48.604477 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10992f8-779b-46dd-9352-b62563ec16c8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:48.604764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10992f8-779b-46dd-9352-b62563ec16c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:48.605026 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10992f8-779b-46dd-9352-b62563ec16c8 admin admin] PATH_INFO: `/v3/projects/3be8e37e626c4f7f88ab9d76ec333d63/users/5a6b59c2ff2148aa9ecc0fcf14453490/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:48.622805 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10992f8-779b-46dd-9352-b62563ec16c8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3be8e37e626c4f7f88ab9d76ec333d63, user_id=5a6b59c2ff2148aa9ecc0fcf14453490, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:48.625524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10992f8-779b-46dd-9352-b62563ec16c8 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:48.638533 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1092/2174] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:48 2022] PUT /identity/v3/projects/3be8e37e626c4f7f88ab9d76ec333d63/users/5a6b59c2ff2148aa9ecc0fcf14453490/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:48.648657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe9bd92-a84f-4b93-b339-f94aa86b5cb7 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:48.648991 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe9bd92-a84f-4b93-b339-f94aa86b5cb7 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:48.649289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe9bd92-a84f-4b93-b339-f94aa86b5cb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:48.679858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-0fe9bd92-a84f-4b93-b339-f94aa86b5cb7 None None] MFA Rules not processed for user `5a6b59c2ff2148aa9ecc0fcf14453490`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:48.681085 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-0fe9bd92-a84f-4b93-b339-f94aa86b5cb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:48.748905 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1083/2175] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:48 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:48.753377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10992f8-779b-46dd-9352-b62563ec16c8 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:48.789257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:48.820991 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10992f8-779b-46dd-9352-b62563ec16c8 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:48.825981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cdee62-d4a4-45a1-ab7e-ea388081f598 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:48.826616 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cdee62-d4a4-45a1-ab7e-ea388081f598 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:48.826902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cdee62-d4a4-45a1-ab7e-ea388081f598 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:48.827214 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cdee62-d4a4-45a1-ab7e-ea388081f598 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:48.827881 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6cdee62-d4a4-45a1-ab7e-ea388081f598 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:48.829823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6cdee62-d4a4-45a1-ab7e-ea388081f598 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:48.862487 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1093/2176] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:48 2022] POST /identity/v3/projects => generated 383 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:48.867379 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe9bd92-a84f-4b93-b339-f94aa86b5cb7 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:48.878280 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe9bd92-a84f-4b93-b339-f94aa86b5cb7 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:48.937803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2624fbc-acbc-46e7-85f1-27479459dfa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:48.938807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2624fbc-acbc-46e7-85f1-27479459dfa3 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:48.939087 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2624fbc-acbc-46e7-85f1-27479459dfa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:48.939400 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2624fbc-acbc-46e7-85f1-27479459dfa3 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:48.940179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2624fbc-acbc-46e7-85f1-27479459dfa3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:48.942221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2624fbc-acbc-46e7-85f1-27479459dfa3 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:48.967888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1084/2177] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:48 2022] POST /identity/v3/users => generated 292 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:58:48.974268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cdee62-d4a4-45a1-ab7e-ea388081f598 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:48.986337 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cdee62-d4a4-45a1-ab7e-ea388081f598 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:48.992433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a671a5-08e7-4e46-8b81-4eebf9d2663b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:48.993232 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a671a5-08e7-4e46-8b81-4eebf9d2663b admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:48.993707 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a671a5-08e7-4e46-8b81-4eebf9d2663b admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:48.993873 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a671a5-08e7-4e46-8b81-4eebf9d2663b admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:48.994495 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a671a5-08e7-4e46-8b81-4eebf9d2663b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:48.996235 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a671a5-08e7-4e46-8b81-4eebf9d2663b admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:49.015477 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1094/2178] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:48 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:49.019748 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2624fbc-acbc-46e7-85f1-27479459dfa3 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:49.038002 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2624fbc-acbc-46e7-85f1-27479459dfa3 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:49.047243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bfee1a-82dd-4db1-ba4c-dcf529abea85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:49.048319 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bfee1a-82dd-4db1-ba4c-dcf529abea85 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:49.049025 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bfee1a-82dd-4db1-ba4c-dcf529abea85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:49.049424 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bfee1a-82dd-4db1-ba4c-dcf529abea85 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:49.050657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46bfee1a-82dd-4db1-ba4c-dcf529abea85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:49.056755 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46bfee1a-82dd-4db1-ba4c-dcf529abea85 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:49.068740 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1085/2179] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:49 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:49.075015 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a671a5-08e7-4e46-8b81-4eebf9d2663b admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:49.087124 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a671a5-08e7-4e46-8b81-4eebf9d2663b admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:49.093190 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c0d594-dfb2-4157-8c74-26f063c7079b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:49.093931 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c0d594-dfb2-4157-8c74-26f063c7079b admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:49.094245 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c0d594-dfb2-4157-8c74-26f063c7079b admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:49.094546 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c0d594-dfb2-4157-8c74-26f063c7079b admin admin] PATH_INFO: `/v3/projects/3c1d9ae2e71b4dd4b261c7278f3f4356/users/1d46da9c132e4a16be409c85798b6ed6/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:49.112825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c0d594-dfb2-4157-8c74-26f063c7079b admin admin] RBAC: Authorizing `identity:create_grant(project_id=3c1d9ae2e71b4dd4b261c7278f3f4356, user_id=1d46da9c132e4a16be409c85798b6ed6, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:49.115628 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c0d594-dfb2-4157-8c74-26f063c7079b admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:49.135592 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1095/2180] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:49 2022] PUT /identity/v3/projects/3c1d9ae2e71b4dd4b261c7278f3f4356/users/1d46da9c132e4a16be409c85798b6ed6/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:49.161408 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee59384b-64ec-43d5-b9c8-c5109d2d6696 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:49.161761 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee59384b-64ec-43d5-b9c8-c5109d2d6696 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:49.161981 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee59384b-64ec-43d5-b9c8-c5109d2d6696 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:49.193800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-ee59384b-64ec-43d5-b9c8-c5109d2d6696 None None] MFA Rules not processed for user `1d46da9c132e4a16be409c85798b6ed6`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:49.196273 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ee59384b-64ec-43d5-b9c8-c5109d2d6696 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:49.278254 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:49.316467 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1086/2181] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:49 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:49.324404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c0d594-dfb2-4157-8c74-26f063c7079b admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:49.419191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c0d594-dfb2-4157-8c74-26f063c7079b admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:49.424153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c01c588-5344-40d8-b310-201172c03769 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:49.424757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c01c588-5344-40d8-b310-201172c03769 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:49.425035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c01c588-5344-40d8-b310-201172c03769 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:49.425312 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c01c588-5344-40d8-b310-201172c03769 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:49.425944 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c01c588-5344-40d8-b310-201172c03769 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:49.427899 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c01c588-5344-40d8-b310-201172c03769 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:49.457004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1096/2182] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:49 2022] POST /identity/v3/projects => generated 385 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:49.462016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee59384b-64ec-43d5-b9c8-c5109d2d6696 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:49.473474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee59384b-64ec-43d5-b9c8-c5109d2d6696 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:49.568796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24eedfe4-6d6e-4080-af0a-594cea4e1789 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:49.568796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24eedfe4-6d6e-4080-af0a-594cea4e1789 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:49.568796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24eedfe4-6d6e-4080-af0a-594cea4e1789 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:49.568796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24eedfe4-6d6e-4080-af0a-594cea4e1789 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:49.569154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24eedfe4-6d6e-4080-af0a-594cea4e1789 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:49.569943 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24eedfe4-6d6e-4080-af0a-594cea4e1789 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:49.615100 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1087/2183] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:49 2022] POST /identity/v3/users => generated 299 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:49.624180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c01c588-5344-40d8-b310-201172c03769 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:49.642343 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c01c588-5344-40d8-b310-201172c03769 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:49.648832 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24eedfe4-6d6e-4080-af0a-594cea4e1789 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:49.651523 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-24eedfe4-6d6e-4080-af0a-594cea4e1789 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:49.657152 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d40dab-dfc0-432f-9b20-68d27bd0e9a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:49.657882 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d40dab-dfc0-432f-9b20-68d27bd0e9a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:49.658344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d40dab-dfc0-432f-9b20-68d27bd0e9a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:49.660217 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d40dab-dfc0-432f-9b20-68d27bd0e9a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:49.661191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d40dab-dfc0-432f-9b20-68d27bd0e9a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:49.667467 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d40dab-dfc0-432f-9b20-68d27bd0e9a4 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:49.679891 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24eedfe4-6d6e-4080-af0a-594cea4e1789 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:49.690670 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1097/2184] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:49 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:49.697901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d40dab-dfc0-432f-9b20-68d27bd0e9a4 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:49.702776 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1728c0-962b-457b-8213-06c56fdd8fd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:49.703696 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1728c0-962b-457b-8213-06c56fdd8fd3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:49.703876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1728c0-962b-457b-8213-06c56fdd8fd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:49.704258 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1728c0-962b-457b-8213-06c56fdd8fd3 admin admin] PATH_INFO: `/v3/users/6defda3dae6f4c1f919bdfecb8e66f26` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:49.705902 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a1728c0-962b-457b-8213-06c56fdd8fd3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6defda3dae6f4c1f919bdfecb8e66f26)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:49.707955 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a1728c0-962b-457b-8213-06c56fdd8fd3 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:49.715457 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d40dab-dfc0-432f-9b20-68d27bd0e9a4 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:49.725974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430b538a-75af-458d-8393-a7734eaaf51e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:49.726953 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430b538a-75af-458d-8393-a7734eaaf51e admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:49.727235 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430b538a-75af-458d-8393-a7734eaaf51e admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:49.727779 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430b538a-75af-458d-8393-a7734eaaf51e admin admin] PATH_INFO: `/v3/projects/369019ee55de44058c3d0c879f7fe185/users/499931665a114e23ba48974d6c5c755d/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:49.753180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430b538a-75af-458d-8393-a7734eaaf51e admin admin] RBAC: Authorizing `identity:create_grant(project_id=369019ee55de44058c3d0c879f7fe185, user_id=499931665a114e23ba48974d6c5c755d, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:49.759471 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430b538a-75af-458d-8393-a7734eaaf51e admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:49.778422 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1098/2185] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:49 2022] PUT /identity/v3/projects/369019ee55de44058c3d0c879f7fe185/users/499931665a114e23ba48974d6c5c755d/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:49.783393 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430b538a-75af-458d-8393-a7734eaaf51e admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:49.801557 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-1a1728c0-962b-457b-8213-06c56fdd8fd3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6defda3dae6f4c1f919bdfecb8e66f26'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:49.816319 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-1a1728c0-962b-457b-8213-06c56fdd8fd3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6defda3dae6f4c1f919bdfecb8e66f26'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:49.817155 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-1a1728c0-962b-457b-8213-06c56fdd8fd3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6defda3dae6f4c1f919bdfecb8e66f26'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:49.818189 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-1a1728c0-962b-457b-8213-06c56fdd8fd3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6defda3dae6f4c1f919bdfecb8e66f26'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:49.826604 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-1a1728c0-962b-457b-8213-06c56fdd8fd3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6defda3dae6f4c1f919bdfecb8e66f26'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:49.837473 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:49.849664 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1088/2186] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:58:49 2022] DELETE /identity/v3/users/6defda3dae6f4c1f919bdfecb8e66f26 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:49.854693 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5183a2a7-6a81-45cb-9c1b-960f6bb93537 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:49.855084 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5183a2a7-6a81-45cb-9c1b-960f6bb93537 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:49.855448 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5183a2a7-6a81-45cb-9c1b-960f6bb93537 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:49.871421 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430b538a-75af-458d-8393-a7734eaaf51e admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:49.876073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-5183a2a7-6a81-45cb-9c1b-960f6bb93537 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:49.877339 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-5183a2a7-6a81-45cb-9c1b-960f6bb93537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:49.928815 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:49.945052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf401b5-03b4-4a9e-b2b3-49ece3f4ea17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:49.945694 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf401b5-03b4-4a9e-b2b3-49ece3f4ea17 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:49.945978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf401b5-03b4-4a9e-b2b3-49ece3f4ea17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:49.946248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf401b5-03b4-4a9e-b2b3-49ece3f4ea17 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:49.946950 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf401b5-03b4-4a9e-b2b3-49ece3f4ea17 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:49.948796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf401b5-03b4-4a9e-b2b3-49ece3f4ea17 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:49.957037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1089/2187] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:58:49.963102 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1099/2188] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:49 2022] GET /identity/v3/roles => generated 1420 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:58:49.964529 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5183a2a7-6a81-45cb-9c1b-960f6bb93537 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:49.966100 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-5183a2a7-6a81-45cb-9c1b-960f6bb93537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:49.968211 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf401b5-03b4-4a9e-b2b3-49ece3f4ea17 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:49.969899 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-ecf401b5-03b4-4a9e-b2b3-49ece3f4ea17 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:49.980201 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5183a2a7-6a81-45cb-9c1b-960f6bb93537 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:49.985361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5581852f-62a1-494f-a174-0f51d8d3c181 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:49.986030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5581852f-62a1-494f-a174-0f51d8d3c181 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:49.986315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5581852f-62a1-494f-a174-0f51d8d3c181 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:49.986624 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5581852f-62a1-494f-a174-0f51d8d3c181 admin admin] PATH_INFO: `/v3/projects/369019ee55de44058c3d0c879f7fe185/users/499931665a114e23ba48974d6c5c755d/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:49.988718 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5581852f-62a1-494f-a174-0f51d8d3c181 admin admin] RBAC: Authorizing `identity:create_grant(project_id=369019ee55de44058c3d0c879f7fe185, user_id=499931665a114e23ba48974d6c5c755d, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:49.991478 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5581852f-62a1-494f-a174-0f51d8d3c181 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:50.002702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1090/2189] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:49 2022] PUT /identity/v3/projects/369019ee55de44058c3d0c879f7fe185/users/499931665a114e23ba48974d6c5c755d/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:50.044220 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf401b5-03b4-4a9e-b2b3-49ece3f4ea17 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:50.075582 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bb35f8-5293-49bc-a376-fbc6d1e78965 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:50.075869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bb35f8-5293-49bc-a376-fbc6d1e78965 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:50.076142 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bb35f8-5293-49bc-a376-fbc6d1e78965 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:50.099672 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9897bc13-efd7-4286-a5f8-91174088eb5c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:50.100292 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9897bc13-efd7-4286-a5f8-91174088eb5c service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:50.100631 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9897bc13-efd7-4286-a5f8-91174088eb5c service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:50.100915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9897bc13-efd7-4286-a5f8-91174088eb5c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:50.108153 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-e9bb35f8-5293-49bc-a376-fbc6d1e78965 None None] MFA Rules not processed for user `499931665a114e23ba48974d6c5c755d`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:50.109390 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-e9bb35f8-5293-49bc-a376-fbc6d1e78965 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:50.148218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:50.166433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9897bc13-efd7-4286-a5f8-91174088eb5c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:50.168052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9897bc13-efd7-4286-a5f8-91174088eb5c service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:50.178343 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1100/2190] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 211 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:58:50.189166 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1091/2191] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:50 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:58:50.204080 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9897bc13-efd7-4286-a5f8-91174088eb5c service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:50.205623 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-9897bc13-efd7-4286-a5f8-91174088eb5c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:50.277946 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9897bc13-efd7-4286-a5f8-91174088eb5c service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:50.284990 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0703d7-225e-4ea5-aa66-c13f36a6cf77 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:50.286062 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0703d7-225e-4ea5-aa66-c13f36a6cf77 service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:50.286569 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0703d7-225e-4ea5-aa66-c13f36a6cf77 service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:50.287030 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0703d7-225e-4ea5-aa66-c13f36a6cf77 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:50.290207 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-5b0703d7-225e-4ea5-aa66-c13f36a6cf77 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:50.366783 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0703d7-225e-4ea5-aa66-c13f36a6cf77 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:50.368234 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0703d7-225e-4ea5-aa66-c13f36a6cf77 service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:50.378425 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1101/2192] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:50 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:50.566826 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bb35f8-5293-49bc-a376-fbc6d1e78965 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:50.569510 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-e9bb35f8-5293-49bc-a376-fbc6d1e78965 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:50.585985 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bb35f8-5293-49bc-a376-fbc6d1e78965 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:50.591444 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bca7ba-66d8-4058-8a0e-cec9d19dfe25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:50.592161 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bca7ba-66d8-4058-8a0e-cec9d19dfe25 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:50.592857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bca7ba-66d8-4058-8a0e-cec9d19dfe25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:50.593236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bca7ba-66d8-4058-8a0e-cec9d19dfe25 admin admin] PATH_INFO: `/v3/projects/716713d46b3743b49e9cbe9949a8c3de` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:50.595024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bca7ba-66d8-4058-8a0e-cec9d19dfe25 admin admin] RBAC: Authorizing `identity:delete_project(project_id=716713d46b3743b49e9cbe9949a8c3de)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:50.597096 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bca7ba-66d8-4058-8a0e-cec9d19dfe25 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:50.652238 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c3bca7ba-66d8-4058-8a0e-cec9d19dfe25 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '716713d46b3743b49e9cbe9949a8c3de', 'request_id': 'req-c3bca7ba-66d8-4058-8a0e-cec9d19dfe25'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:50.658820 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c3bca7ba-66d8-4058-8a0e-cec9d19dfe25 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '716713d46b3743b49e9cbe9949a8c3de', 'request_id': 'req-c3bca7ba-66d8-4058-8a0e-cec9d19dfe25'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:50.668114 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c3bca7ba-66d8-4058-8a0e-cec9d19dfe25 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '716713d46b3743b49e9cbe9949a8c3de', 'request_id': 'req-c3bca7ba-66d8-4058-8a0e-cec9d19dfe25'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:50.690900 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c3bca7ba-66d8-4058-8a0e-cec9d19dfe25 admin admin] The token cache is being invalidate because project 716713d46b3743b49e9cbe9949a8c3de was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:58:50.691682 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c3bca7ba-66d8-4058-8a0e-cec9d19dfe25 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:50.692808 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c3bca7ba-66d8-4058-8a0e-cec9d19dfe25 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:58:50.703429 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1092/2193] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:58:50 2022] DELETE /identity/v3/projects/716713d46b3743b49e9cbe9949a8c3de => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:50.703930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3561940a-8e98-4885-b9cc-b4ff8c80c217 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:50.704215 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3561940a-8e98-4885-b9cc-b4ff8c80c217 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:50.704540 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3561940a-8e98-4885-b9cc-b4ff8c80c217 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:50.729234 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-3561940a-8e98-4885-b9cc-b4ff8c80c217 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:50.747864 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3561940a-8e98-4885-b9cc-b4ff8c80c217 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:50.841945 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1102/2194] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:50.851256 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73a0eea-cb76-41e9-a19d-65aff6034906 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:50.851625 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73a0eea-cb76-41e9-a19d-65aff6034906 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:50.852072 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73a0eea-cb76-41e9-a19d-65aff6034906 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:50.877295 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-c73a0eea-cb76-41e9-a19d-65aff6034906 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:50.879724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c73a0eea-cb76-41e9-a19d-65aff6034906 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:50.890500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1093/2195] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:58:50.894537 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3561940a-8e98-4885-b9cc-b4ff8c80c217 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:50.908019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3561940a-8e98-4885-b9cc-b4ff8c80c217 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:50.913747 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f436be-3990-4a1f-90ee-bd28d71cb898 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:50.915116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f436be-3990-4a1f-90ee-bd28d71cb898 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:50.915116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f436be-3990-4a1f-90ee-bd28d71cb898 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:50.915531 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f436be-3990-4a1f-90ee-bd28d71cb898 admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:50.916486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f436be-3990-4a1f-90ee-bd28d71cb898 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:58:50.916941 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f436be-3990-4a1f-90ee-bd28d71cb898 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:50.918751 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f436be-3990-4a1f-90ee-bd28d71cb898 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:50.932222 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1103/2196] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:58:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:58:50.936542 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73a0eea-cb76-41e9-a19d-65aff6034906 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:50.946957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73a0eea-cb76-41e9-a19d-65aff6034906 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:50.952222 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d20f24d-b214-4a55-a6fd-cfcc2015936e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:50.952837 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d20f24d-b214-4a55-a6fd-cfcc2015936e admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:50.953110 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d20f24d-b214-4a55-a6fd-cfcc2015936e admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:50.953376 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d20f24d-b214-4a55-a6fd-cfcc2015936e admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:50.954020 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d20f24d-b214-4a55-a6fd-cfcc2015936e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:50.956011 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d20f24d-b214-4a55-a6fd-cfcc2015936e admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:50.986514 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1094/2197] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:58:50 2022] POST /identity/v3/projects => generated 369 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:58:50.990149 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f436be-3990-4a1f-90ee-bd28d71cb898 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:51.000792 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f436be-3990-4a1f-90ee-bd28d71cb898 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:51.026386 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:51.064471 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee716e6b-1ea5-49ab-a7a2-c50440ea9923 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:51.065084 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee716e6b-1ea5-49ab-a7a2-c50440ea9923 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:51.065426 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee716e6b-1ea5-49ab-a7a2-c50440ea9923 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:51.065647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee716e6b-1ea5-49ab-a7a2-c50440ea9923 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:51.066298 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee716e6b-1ea5-49ab-a7a2-c50440ea9923 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:51.068275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee716e6b-1ea5-49ab-a7a2-c50440ea9923 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:51.092199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1104/2198] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:58:50 2022] POST /identity/v3/users => generated 285 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:58:51.096199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d20f24d-b214-4a55-a6fd-cfcc2015936e admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:51.106807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d20f24d-b214-4a55-a6fd-cfcc2015936e admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:51.111492 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2797b90a-c0ee-4c5e-a4c4-98ad60c9c0c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:51.112092 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2797b90a-c0ee-4c5e-a4c4-98ad60c9c0c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:51.112397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2797b90a-c0ee-4c5e-a4c4-98ad60c9c0c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:51.112668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2797b90a-c0ee-4c5e-a4c4-98ad60c9c0c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:51.113310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2797b90a-c0ee-4c5e-a4c4-98ad60c9c0c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:51.115459 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2797b90a-c0ee-4c5e-a4c4-98ad60c9c0c0 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:51.120985 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:51.127144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1095/2199] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:51.135357 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee716e6b-1ea5-49ab-a7a2-c50440ea9923 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:51.145287 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee716e6b-1ea5-49ab-a7a2-c50440ea9923 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:51.149968 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8090c3-09e0-45f1-9be0-92e897f9817a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:51.150612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8090c3-09e0-45f1-9be0-92e897f9817a admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:51.150898 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8090c3-09e0-45f1-9be0-92e897f9817a admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:51.151179 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8090c3-09e0-45f1-9be0-92e897f9817a admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:51.151888 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8090c3-09e0-45f1-9be0-92e897f9817a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:51.153602 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8090c3-09e0-45f1-9be0-92e897f9817a admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:51.165174 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1105/2200] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:58:51 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:58:51.169373 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2797b90a-c0ee-4c5e-a4c4-98ad60c9c0c0 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:51.180220 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2797b90a-c0ee-4c5e-a4c4-98ad60c9c0c0 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:51.185038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbeda59-464e-44fb-8ddf-591305521028 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:51.185689 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbeda59-464e-44fb-8ddf-591305521028 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:51.185962 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbeda59-464e-44fb-8ddf-591305521028 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:51.186256 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbeda59-464e-44fb-8ddf-591305521028 admin admin] PATH_INFO: `/v3/projects/faf829449e074324901c08d9e1215533/users/c9f9871f4c3e4a2c836e27edd2a52558/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:51.204939 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecbeda59-464e-44fb-8ddf-591305521028 admin admin] RBAC: Authorizing `identity:create_grant(project_id=faf829449e074324901c08d9e1215533, user_id=c9f9871f4c3e4a2c836e27edd2a52558, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:51.207740 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecbeda59-464e-44fb-8ddf-591305521028 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:51.221381 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1096/2201] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:58:51 2022] PUT /identity/v3/projects/faf829449e074324901c08d9e1215533/users/c9f9871f4c3e4a2c836e27edd2a52558/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:58:51.231850 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b7b496-21de-4d52-b728-3a4f8504f54e None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:51.232140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b7b496-21de-4d52-b728-3a4f8504f54e None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:51.232418 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b7b496-21de-4d52-b728-3a4f8504f54e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:51.262179 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-48b7b496-21de-4d52-b728-3a4f8504f54e None None] MFA Rules not processed for user `c9f9871f4c3e4a2c836e27edd2a52558`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:58:51.263580 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-48b7b496-21de-4d52-b728-3a4f8504f54e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:51.332224 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1106/2202] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:58:51 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:58:51.344738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbeda59-464e-44fb-8ddf-591305521028 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:51.346530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ecbeda59-464e-44fb-8ddf-591305521028 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:51.410747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbeda59-464e-44fb-8ddf-591305521028 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:51.415355 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3c664c-58eb-4167-8da5-676c3077c867 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7fe9d21099e4c4891c28687bb673214', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:51.415970 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3c664c-58eb-4167-8da5-676c3077c867 service glance] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:51.416292 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3c664c-58eb-4167-8da5-676c3077c867 service glance] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:51.416551 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3c664c-58eb-4167-8da5-676c3077c867 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:51.426121 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3c664c-58eb-4167-8da5-676c3077c867 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:51.428209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3c664c-58eb-4167-8da5-676c3077c867 service glance] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:51.438304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1097/2203] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:51 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:55.234172 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b7b496-21de-4d52-b728-3a4f8504f54e None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:55.236229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-48b7b496-21de-4d52-b728-3a4f8504f54e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:55.303785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b7b496-21de-4d52-b728-3a4f8504f54e None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:55.320507 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb331729-477b-4806-a808-5aecd8b50dc1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:55.321214 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb331729-477b-4806-a808-5aecd8b50dc1 service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:55.321723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb331729-477b-4806-a808-5aecd8b50dc1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:55.322208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb331729-477b-4806-a808-5aecd8b50dc1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:55.324411 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-fb331729-477b-4806-a808-5aecd8b50dc1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:55.374604 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:58:55.391471 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb331729-477b-4806-a808-5aecd8b50dc1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:55.392912 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb331729-477b-4806-a808-5aecd8b50dc1 service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:55.402947 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1107/2204] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:55 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 169 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:58:56.014811 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3c664c-58eb-4167-8da5-676c3077c867 service glance] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:58:56.025013 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3c664c-58eb-4167-8da5-676c3077c867 service glance] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:58:56.028780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982e617a-d8b8-471e-82d5-39427b0f27bb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:58:56.029380 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982e617a-d8b8-471e-82d5-39427b0f27bb service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:58:56.029657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982e617a-d8b8-471e-82d5-39427b0f27bb service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:58:56.029927 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982e617a-d8b8-471e-82d5-39427b0f27bb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:58:56.031839 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-982e617a-d8b8-471e-82d5-39427b0f27bb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:58:56.101464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982e617a-d8b8-471e-82d5-39427b0f27bb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:58:56.103040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982e617a-d8b8-471e-82d5-39427b0f27bb service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:58:56.112784 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1098/2205] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:58:56 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:59:02.268074 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb331729-477b-4806-a808-5aecd8b50dc1 service cinder] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:02.269653 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-fb331729-477b-4806-a808-5aecd8b50dc1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:02.368651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb331729-477b-4806-a808-5aecd8b50dc1 service cinder] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:02.372906 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cbbf8e-748b-40fd-b3ac-757a5043c1b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:02.373656 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cbbf8e-748b-40fd-b3ac-757a5043c1b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:02.374006 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cbbf8e-748b-40fd-b3ac-757a5043c1b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:02.374352 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cbbf8e-748b-40fd-b3ac-757a5043c1b3 admin admin] PATH_INFO: `/v3/users/5a6b59c2ff2148aa9ecc0fcf14453490` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:02.375603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cbbf8e-748b-40fd-b3ac-757a5043c1b3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5a6b59c2ff2148aa9ecc0fcf14453490)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:02.377554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cbbf8e-748b-40fd-b3ac-757a5043c1b3 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:02.427759 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-87cbbf8e-748b-40fd-b3ac-757a5043c1b3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5a6b59c2ff2148aa9ecc0fcf14453490'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:02.438589 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-87cbbf8e-748b-40fd-b3ac-757a5043c1b3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5a6b59c2ff2148aa9ecc0fcf14453490'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:02.452393 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-87cbbf8e-748b-40fd-b3ac-757a5043c1b3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5a6b59c2ff2148aa9ecc0fcf14453490'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:02.453308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-87cbbf8e-748b-40fd-b3ac-757a5043c1b3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5a6b59c2ff2148aa9ecc0fcf14453490'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:02.460836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-87cbbf8e-748b-40fd-b3ac-757a5043c1b3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5a6b59c2ff2148aa9ecc0fcf14453490'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:02.466877 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1108/2206] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:59:02 2022] DELETE /identity/v3/users/5a6b59c2ff2148aa9ecc0fcf14453490 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:02.471513 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92051167-c9f9-4afd-a908-3cccc71cd062 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:02.472004 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92051167-c9f9-4afd-a908-3cccc71cd062 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:02.472153 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92051167-c9f9-4afd-a908-3cccc71cd062 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:02.493587 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-92051167-c9f9-4afd-a908-3cccc71cd062 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:02.495766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-92051167-c9f9-4afd-a908-3cccc71cd062 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:02.565440 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1099/2207] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:02.577117 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cbbf8e-748b-40fd-b3ac-757a5043c1b3 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:02.578663 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-87cbbf8e-748b-40fd-b3ac-757a5043c1b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:02.646153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cbbf8e-748b-40fd-b3ac-757a5043c1b3 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:02.650827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bff3ef-ef0b-45c5-ba83-d3368c68bf72 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:02.651671 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bff3ef-ef0b-45c5-ba83-d3368c68bf72 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:02.651955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bff3ef-ef0b-45c5-ba83-d3368c68bf72 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:02.652233 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bff3ef-ef0b-45c5-ba83-d3368c68bf72 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:02.664142 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bff3ef-ef0b-45c5-ba83-d3368c68bf72 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:02.665644 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bff3ef-ef0b-45c5-ba83-d3368c68bf72 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:02.677774 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1109/2208] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:59:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:59:02.971893 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92051167-c9f9-4afd-a908-3cccc71cd062 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:02.973367 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-92051167-c9f9-4afd-a908-3cccc71cd062 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:02.987164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92051167-c9f9-4afd-a908-3cccc71cd062 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:02.992131 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4251bdc9-71ea-4fa4-bff3-2893465870de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:02.992881 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4251bdc9-71ea-4fa4-bff3-2893465870de admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:02.993161 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4251bdc9-71ea-4fa4-bff3-2893465870de admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:02.993427 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4251bdc9-71ea-4fa4-bff3-2893465870de admin admin] PATH_INFO: `/v3/projects/3be8e37e626c4f7f88ab9d76ec333d63` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:02.994636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4251bdc9-71ea-4fa4-bff3-2893465870de admin admin] RBAC: Authorizing `identity:delete_project(project_id=3be8e37e626c4f7f88ab9d76ec333d63)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:02.996602 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4251bdc9-71ea-4fa4-bff3-2893465870de admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:03.006166 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:03.058978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-4251bdc9-71ea-4fa4-bff3-2893465870de admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3be8e37e626c4f7f88ab9d76ec333d63', 'request_id': 'req-4251bdc9-71ea-4fa4-bff3-2893465870de'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:03.064674 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-4251bdc9-71ea-4fa4-bff3-2893465870de admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3be8e37e626c4f7f88ab9d76ec333d63', 'request_id': 'req-4251bdc9-71ea-4fa4-bff3-2893465870de'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:03.074212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-4251bdc9-71ea-4fa4-bff3-2893465870de admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3be8e37e626c4f7f88ab9d76ec333d63', 'request_id': 'req-4251bdc9-71ea-4fa4-bff3-2893465870de'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:03.086254 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-4251bdc9-71ea-4fa4-bff3-2893465870de admin admin] The token cache is being invalidate because project 3be8e37e626c4f7f88ab9d76ec333d63 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:59:03.086569 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-4251bdc9-71ea-4fa4-bff3-2893465870de admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:03.087215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-4251bdc9-71ea-4fa4-bff3-2893465870de admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:03.088703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1100/2209] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:59:02 2022] DELETE /identity/v3/projects/3be8e37e626c4f7f88ab9d76ec333d63 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:03.093319 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bff3ef-ef0b-45c5-ba83-d3368c68bf72 service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:03.105090 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bff3ef-ef0b-45c5-ba83-d3368c68bf72 service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:03.117679 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:03.181144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a13a91-0e4c-41e1-bcab-d720a7ef5f47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:03.181787 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a13a91-0e4c-41e1-bcab-d720a7ef5f47 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:03.182068 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a13a91-0e4c-41e1-bcab-d720a7ef5f47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:03.182342 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a13a91-0e4c-41e1-bcab-d720a7ef5f47 admin admin] PATH_INFO: `/v3/users/1d46da9c132e4a16be409c85798b6ed6` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:03.183519 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a13a91-0e4c-41e1-bcab-d720a7ef5f47 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1d46da9c132e4a16be409c85798b6ed6)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:03.185477 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a13a91-0e4c-41e1-bcab-d720a7ef5f47 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:03.236030 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-e6a13a91-0e4c-41e1-bcab-d720a7ef5f47 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1d46da9c132e4a16be409c85798b6ed6'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:03.245152 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-e6a13a91-0e4c-41e1-bcab-d720a7ef5f47 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1d46da9c132e4a16be409c85798b6ed6'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:03.259240 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-e6a13a91-0e4c-41e1-bcab-d720a7ef5f47 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1d46da9c132e4a16be409c85798b6ed6'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:03.260208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-e6a13a91-0e4c-41e1-bcab-d720a7ef5f47 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1d46da9c132e4a16be409c85798b6ed6'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:03.267297 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-e6a13a91-0e4c-41e1-bcab-d720a7ef5f47 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1d46da9c132e4a16be409c85798b6ed6'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:03.277178 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1110/2210] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:59:03 2022] DELETE /identity/v3/users/1d46da9c132e4a16be409c85798b6ed6 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:03.607092 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4251bdc9-71ea-4fa4-bff3-2893465870de admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:03.609266 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-4251bdc9-71ea-4fa4-bff3-2893465870de admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:03.628085 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4251bdc9-71ea-4fa4-bff3-2893465870de admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:03.693913 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b72253-e8ab-4a45-9bd0-b8007c005250 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:03.694578 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b72253-e8ab-4a45-9bd0-b8007c005250 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:03.694856 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b72253-e8ab-4a45-9bd0-b8007c005250 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:03.695127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b72253-e8ab-4a45-9bd0-b8007c005250 admin admin] PATH_INFO: `/v3/projects/3c1d9ae2e71b4dd4b261c7278f3f4356` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:03.696258 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b72253-e8ab-4a45-9bd0-b8007c005250 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3c1d9ae2e71b4dd4b261c7278f3f4356)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:03.698191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b72253-e8ab-4a45-9bd0-b8007c005250 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:03.749168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-72b72253-e8ab-4a45-9bd0-b8007c005250 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3c1d9ae2e71b4dd4b261c7278f3f4356', 'request_id': 'req-72b72253-e8ab-4a45-9bd0-b8007c005250'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:03.754615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-72b72253-e8ab-4a45-9bd0-b8007c005250 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3c1d9ae2e71b4dd4b261c7278f3f4356', 'request_id': 'req-72b72253-e8ab-4a45-9bd0-b8007c005250'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:03.763397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-72b72253-e8ab-4a45-9bd0-b8007c005250 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3c1d9ae2e71b4dd4b261c7278f3f4356', 'request_id': 'req-72b72253-e8ab-4a45-9bd0-b8007c005250'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:03.776574 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-72b72253-e8ab-4a45-9bd0-b8007c005250 admin admin] The token cache is being invalidate because project 3c1d9ae2e71b4dd4b261c7278f3f4356 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:59:03.776859 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-72b72253-e8ab-4a45-9bd0-b8007c005250 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:03.777517 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-72b72253-e8ab-4a45-9bd0-b8007c005250 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:03.779031 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1101/2211] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:59:03 2022] DELETE /identity/v3/projects/3c1d9ae2e71b4dd4b261c7278f3f4356 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:03.782998 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a13a91-0e4c-41e1-bcab-d720a7ef5f47 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:03.784450 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-e6a13a91-0e4c-41e1-bcab-d720a7ef5f47 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:03.797597 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a13a91-0e4c-41e1-bcab-d720a7ef5f47 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:03.855984 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf1b56d-4ad4-44de-95ac-49b6bb7f7f78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:03.856627 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf1b56d-4ad4-44de-95ac-49b6bb7f7f78 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:03.856908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf1b56d-4ad4-44de-95ac-49b6bb7f7f78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:03.857184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf1b56d-4ad4-44de-95ac-49b6bb7f7f78 admin admin] PATH_INFO: `/v3/users/499931665a114e23ba48974d6c5c755d` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:03.858327 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf1b56d-4ad4-44de-95ac-49b6bb7f7f78 admin admin] RBAC: Authorizing `identity:delete_user(user_id=499931665a114e23ba48974d6c5c755d)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:03.860297 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf1b56d-4ad4-44de-95ac-49b6bb7f7f78 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:03.894227 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:03.909432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-adf1b56d-4ad4-44de-95ac-49b6bb7f7f78 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '499931665a114e23ba48974d6c5c755d'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:03.916025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-adf1b56d-4ad4-44de-95ac-49b6bb7f7f78 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '499931665a114e23ba48974d6c5c755d'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:03.928658 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-adf1b56d-4ad4-44de-95ac-49b6bb7f7f78 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '499931665a114e23ba48974d6c5c755d'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:03.929386 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-adf1b56d-4ad4-44de-95ac-49b6bb7f7f78 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '499931665a114e23ba48974d6c5c755d'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:03.937398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-adf1b56d-4ad4-44de-95ac-49b6bb7f7f78 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '499931665a114e23ba48974d6c5c755d'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:03.944806 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1111/2212] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:59:03 2022] DELETE /identity/v3/users/499931665a114e23ba48974d6c5c755d => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:04.221577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b72253-e8ab-4a45-9bd0-b8007c005250 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:04.223086 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-72b72253-e8ab-4a45-9bd0-b8007c005250 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:04.236891 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b72253-e8ab-4a45-9bd0-b8007c005250 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:04.261969 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:04.296228 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be84098-b04a-445d-80bb-2d7113b8ea74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:04.296856 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be84098-b04a-445d-80bb-2d7113b8ea74 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:04.297141 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be84098-b04a-445d-80bb-2d7113b8ea74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:04.297416 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be84098-b04a-445d-80bb-2d7113b8ea74 admin admin] PATH_INFO: `/v3/projects/369019ee55de44058c3d0c879f7fe185` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:04.298695 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be84098-b04a-445d-80bb-2d7113b8ea74 admin admin] RBAC: Authorizing `identity:delete_project(project_id=369019ee55de44058c3d0c879f7fe185)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:04.300614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be84098-b04a-445d-80bb-2d7113b8ea74 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:04.346758 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8be84098-b04a-445d-80bb-2d7113b8ea74 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '369019ee55de44058c3d0c879f7fe185', 'request_id': 'req-8be84098-b04a-445d-80bb-2d7113b8ea74'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:04.352821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8be84098-b04a-445d-80bb-2d7113b8ea74 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '369019ee55de44058c3d0c879f7fe185', 'request_id': 'req-8be84098-b04a-445d-80bb-2d7113b8ea74'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:04.360249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8be84098-b04a-445d-80bb-2d7113b8ea74 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '369019ee55de44058c3d0c879f7fe185', 'request_id': 'req-8be84098-b04a-445d-80bb-2d7113b8ea74'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:04.372321 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8be84098-b04a-445d-80bb-2d7113b8ea74 admin admin] The token cache is being invalidate because project 369019ee55de44058c3d0c879f7fe185 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:59:04.372611 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8be84098-b04a-445d-80bb-2d7113b8ea74 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:04.373326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8be84098-b04a-445d-80bb-2d7113b8ea74 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:04.374933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1102/2213] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:59:04 2022] DELETE /identity/v3/projects/369019ee55de44058c3d0c879f7fe185 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:04.380907 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e034d243-f289-4434-b535-ae6a4cd11871 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:04.381473 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e034d243-f289-4434-b535-ae6a4cd11871 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:04.381564 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e034d243-f289-4434-b535-ae6a4cd11871 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:04.400706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-e034d243-f289-4434-b535-ae6a4cd11871 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:04.401898 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-e034d243-f289-4434-b535-ae6a4cd11871 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:04.468046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1112/2214] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:04.476806 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19aaa0c-420c-45cf-8b41-a82a2a3e1280 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:04.477305 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19aaa0c-420c-45cf-8b41-a82a2a3e1280 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:04.477632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19aaa0c-420c-45cf-8b41-a82a2a3e1280 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:04.497827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-b19aaa0c-420c-45cf-8b41-a82a2a3e1280 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:04.499072 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-b19aaa0c-420c-45cf-8b41-a82a2a3e1280 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:04.508942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1103/2215] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:04.513181 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e034d243-f289-4434-b535-ae6a4cd11871 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:04.523672 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e034d243-f289-4434-b535-ae6a4cd11871 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:04.528405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978c32e4-ad0f-40f3-bbe1-2fc5aa769f6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:04.529060 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978c32e4-ad0f-40f3-bbe1-2fc5aa769f6b admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:04.529339 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978c32e4-ad0f-40f3-bbe1-2fc5aa769f6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:04.529637 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978c32e4-ad0f-40f3-bbe1-2fc5aa769f6b admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:04.530293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978c32e4-ad0f-40f3-bbe1-2fc5aa769f6b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:59:04.530620 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978c32e4-ad0f-40f3-bbe1-2fc5aa769f6b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:04.532221 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978c32e4-ad0f-40f3-bbe1-2fc5aa769f6b admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:04.544617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1113/2216] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:59:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:59:04.549027 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19aaa0c-420c-45cf-8b41-a82a2a3e1280 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:04.558135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19aaa0c-420c-45cf-8b41-a82a2a3e1280 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:04.562816 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268a1e12-312c-4eec-ae50-173e06358967 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:04.563404 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268a1e12-312c-4eec-ae50-173e06358967 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:04.563681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268a1e12-312c-4eec-ae50-173e06358967 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:04.563949 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268a1e12-312c-4eec-ae50-173e06358967 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:04.564572 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268a1e12-312c-4eec-ae50-173e06358967 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:04.566450 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268a1e12-312c-4eec-ae50-173e06358967 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:04.594528 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1104/2217] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:59:04 2022] POST /identity/v3/projects => generated 383 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:59:04.598779 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978c32e4-ad0f-40f3-bbe1-2fc5aa769f6b admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:04.609832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978c32e4-ad0f-40f3-bbe1-2fc5aa769f6b admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:04.690093 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8538b4de-c0cb-40e3-a1ca-a3cf1bea8be5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:04.690740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8538b4de-c0cb-40e3-a1ca-a3cf1bea8be5 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:04.691015 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8538b4de-c0cb-40e3-a1ca-a3cf1bea8be5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:04.691285 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8538b4de-c0cb-40e3-a1ca-a3cf1bea8be5 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:04.691939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8538b4de-c0cb-40e3-a1ca-a3cf1bea8be5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:04.693928 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8538b4de-c0cb-40e3-a1ca-a3cf1bea8be5 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:04.722558 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1114/2218] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:59:04 2022] POST /identity/v3/users => generated 292 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:59:04.727278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268a1e12-312c-4eec-ae50-173e06358967 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:04.738348 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268a1e12-312c-4eec-ae50-173e06358967 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:04.743382 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bd49a7-bac8-4b48-a490-a65e3b80c8fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:04.744003 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bd49a7-bac8-4b48-a490-a65e3b80c8fa admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:04.744284 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bd49a7-bac8-4b48-a490-a65e3b80c8fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:04.744559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bd49a7-bac8-4b48-a490-a65e3b80c8fa admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:04.745214 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bd49a7-bac8-4b48-a490-a65e3b80c8fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:04.746905 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bd49a7-bac8-4b48-a490-a65e3b80c8fa admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:04.760094 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1105/2219] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:04 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:59:04.764025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8538b4de-c0cb-40e3-a1ca-a3cf1bea8be5 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:04.774855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8538b4de-c0cb-40e3-a1ca-a3cf1bea8be5 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:04.780019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1feba6f6-91a6-415f-aaf5-0c0822cacc60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:04.780640 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1feba6f6-91a6-415f-aaf5-0c0822cacc60 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:04.780922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1feba6f6-91a6-415f-aaf5-0c0822cacc60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:04.781184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1feba6f6-91a6-415f-aaf5-0c0822cacc60 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:04.781842 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1feba6f6-91a6-415f-aaf5-0c0822cacc60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:04.783469 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1feba6f6-91a6-415f-aaf5-0c0822cacc60 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:04.798054 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1115/2220] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:04 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:59:04.800912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bd49a7-bac8-4b48-a490-a65e3b80c8fa admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:04.811195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bd49a7-bac8-4b48-a490-a65e3b80c8fa admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:04.815883 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-325be6d8-aea0-441a-b47f-93cfb37752f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:04.816526 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325be6d8-aea0-441a-b47f-93cfb37752f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:04.816806 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325be6d8-aea0-441a-b47f-93cfb37752f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:04.817077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325be6d8-aea0-441a-b47f-93cfb37752f3 admin admin] PATH_INFO: `/v3/projects/b1c1956c4ef044c6b2dd7d3df2966ef4/users/c2da7c1e25a541d988f09231e80cd8a9/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:04.833001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-325be6d8-aea0-441a-b47f-93cfb37752f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b1c1956c4ef044c6b2dd7d3df2966ef4, user_id=c2da7c1e25a541d988f09231e80cd8a9, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:04.836703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-325be6d8-aea0-441a-b47f-93cfb37752f3 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:04.851248 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1106/2221] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:59:04 2022] PUT /identity/v3/projects/b1c1956c4ef044c6b2dd7d3df2966ef4/users/c2da7c1e25a541d988f09231e80cd8a9/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:04.861331 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cc11eb-434c-4c2e-b2d2-fc4bea31aff1 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:04.861620 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cc11eb-434c-4c2e-b2d2-fc4bea31aff1 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:04.861898 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cc11eb-434c-4c2e-b2d2-fc4bea31aff1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:04.887092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:04.890405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-a5cc11eb-434c-4c2e-b2d2-fc4bea31aff1 None None] MFA Rules not processed for user `c2da7c1e25a541d988f09231e80cd8a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:04.892310 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a5cc11eb-434c-4c2e-b2d2-fc4bea31aff1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:04.962816 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1116/2222] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:04 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:04.974596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-325be6d8-aea0-441a-b47f-93cfb37752f3 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:04.976306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-325be6d8-aea0-441a-b47f-93cfb37752f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:05.038543 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-325be6d8-aea0-441a-b47f-93cfb37752f3 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:05.043998 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cd16a8-117f-4d4d-9881-1b115e7b8f20 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:05.044767 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cd16a8-117f-4d4d-9881-1b115e7b8f20 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:05.045577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cd16a8-117f-4d4d-9881-1b115e7b8f20 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:05.045577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cd16a8-117f-4d4d-9881-1b115e7b8f20 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:05.052228 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:05.055180 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47cd16a8-117f-4d4d-9881-1b115e7b8f20 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:05.057606 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47cd16a8-117f-4d4d-9881-1b115e7b8f20 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:05.069025 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1107/2223] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:59:04 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:59:07.510071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5cc11eb-434c-4c2e-b2d2-fc4bea31aff1 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:07.511520 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a5cc11eb-434c-4c2e-b2d2-fc4bea31aff1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:07.576125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5cc11eb-434c-4c2e-b2d2-fc4bea31aff1 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:07.580569 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffae1a52-39c0-41b9-9a53-365a53175c78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:07.581290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffae1a52-39c0-41b9-9a53-365a53175c78 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:07.581561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffae1a52-39c0-41b9-9a53-365a53175c78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:07.581852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffae1a52-39c0-41b9-9a53-365a53175c78 admin admin] PATH_INFO: `/v3/users/b274269a7d2a4507b9d08615091d557f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:07.583701 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffae1a52-39c0-41b9-9a53-365a53175c78 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b274269a7d2a4507b9d08615091d557f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:07.585803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffae1a52-39c0-41b9-9a53-365a53175c78 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:07.644187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-ffae1a52-39c0-41b9-9a53-365a53175c78 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b274269a7d2a4507b9d08615091d557f'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:07.655720 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cd16a8-117f-4d4d-9881-1b115e7b8f20 service cinder] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:07.658494 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-ffae1a52-39c0-41b9-9a53-365a53175c78 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b274269a7d2a4507b9d08615091d557f'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:07.670119 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cd16a8-117f-4d4d-9881-1b115e7b8f20 service cinder] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:07.672797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-ffae1a52-39c0-41b9-9a53-365a53175c78 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b274269a7d2a4507b9d08615091d557f'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:07.673492 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-ffae1a52-39c0-41b9-9a53-365a53175c78 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b274269a7d2a4507b9d08615091d557f'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:07.674967 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228f8496-f60c-40b3-99dc-a349a3232ad3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:07.675598 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228f8496-f60c-40b3-99dc-a349a3232ad3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:07.675869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228f8496-f60c-40b3-99dc-a349a3232ad3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:07.676133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228f8496-f60c-40b3-99dc-a349a3232ad3 admin admin] PATH_INFO: `/v3/users/c2da7c1e25a541d988f09231e80cd8a9` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:07.677291 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-228f8496-f60c-40b3-99dc-a349a3232ad3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c2da7c1e25a541d988f09231e80cd8a9)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:07.679306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-228f8496-f60c-40b3-99dc-a349a3232ad3 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:07.680319 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-ffae1a52-39c0-41b9-9a53-365a53175c78 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b274269a7d2a4507b9d08615091d557f'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:07.686614 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1117/2224] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:59:07 2022] DELETE /identity/v3/users/b274269a7d2a4507b9d08615091d557f => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:07.691057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a434f1f-a6ad-40af-92b3-9ab5069c89e8 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:07.691506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a434f1f-a6ad-40af-92b3-9ab5069c89e8 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:07.691780 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a434f1f-a6ad-40af-92b3-9ab5069c89e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:07.711608 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-2a434f1f-a6ad-40af-92b3-9ab5069c89e8 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:07.712880 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-2a434f1f-a6ad-40af-92b3-9ab5069c89e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:07.732049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-228f8496-f60c-40b3-99dc-a349a3232ad3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c2da7c1e25a541d988f09231e80cd8a9'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:07.740679 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:07.742110 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-228f8496-f60c-40b3-99dc-a349a3232ad3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c2da7c1e25a541d988f09231e80cd8a9'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:07.742890 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-228f8496-f60c-40b3-99dc-a349a3232ad3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c2da7c1e25a541d988f09231e80cd8a9'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:07.743864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-228f8496-f60c-40b3-99dc-a349a3232ad3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c2da7c1e25a541d988f09231e80cd8a9'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:07.753728 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-228f8496-f60c-40b3-99dc-a349a3232ad3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c2da7c1e25a541d988f09231e80cd8a9'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:07.780402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1108/2225] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:59:07 2022] DELETE /identity/v3/users/c2da7c1e25a541d988f09231e80cd8a9 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:07.785765 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9bd0ae8-3aa4-4d20-bb64-22c58e84e3c3 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:07.787103 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9bd0ae8-3aa4-4d20-bb64-22c58e84e3c3 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:07.787611 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9bd0ae8-3aa4-4d20-bb64-22c58e84e3c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:07.790504 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1118/2226] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:59:07.800512 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a434f1f-a6ad-40af-92b3-9ab5069c89e8 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:07.802030 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-2a434f1f-a6ad-40af-92b3-9ab5069c89e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:07.820489 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-d9bd0ae8-3aa4-4d20-bb64-22c58e84e3c3 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:07.822280 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d9bd0ae8-3aa4-4d20-bb64-22c58e84e3c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:07.875527 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a434f1f-a6ad-40af-92b3-9ab5069c89e8 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:07.891083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1109/2227] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:59:07.905121 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9bd0ae8-3aa4-4d20-bb64-22c58e84e3c3 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:07.928136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b709ec-82d8-48fd-b6b5-d60a89c822ce service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:07.928528 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b709ec-82d8-48fd-b6b5-d60a89c822ce service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:07.928804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b709ec-82d8-48fd-b6b5-d60a89c822ce service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:07.929077 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b709ec-82d8-48fd-b6b5-d60a89c822ce service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:07.938598 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b709ec-82d8-48fd-b6b5-d60a89c822ce service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:07.940088 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b709ec-82d8-48fd-b6b5-d60a89c822ce service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:07.954210 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1119/2228] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:59:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:59:07.963957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9bd0ae8-3aa4-4d20-bb64-22c58e84e3c3 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:07.967607 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc009368-8268-46c0-86e7-6d66f93048dd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:07.968221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc009368-8268-46c0-86e7-6d66f93048dd service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:07.968499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc009368-8268-46c0-86e7-6d66f93048dd service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:07.968772 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc009368-8268-46c0-86e7-6d66f93048dd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:07.977912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc009368-8268-46c0-86e7-6d66f93048dd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:07.979373 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc009368-8268-46c0-86e7-6d66f93048dd service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:07.992103 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1110/2229] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:59:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:59:08.459934 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b709ec-82d8-48fd-b6b5-d60a89c822ce service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:08.461621 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-38b709ec-82d8-48fd-b6b5-d60a89c822ce service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:08.469572 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc009368-8268-46c0-86e7-6d66f93048dd service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:08.471029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-bc009368-8268-46c0-86e7-6d66f93048dd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:08.481818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b709ec-82d8-48fd-b6b5-d60a89c822ce service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:08.488546 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a317bd6-e136-480b-b436-38c8c62d3ea6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:08.489233 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a317bd6-e136-480b-b436-38c8c62d3ea6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:08.489603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a317bd6-e136-480b-b436-38c8c62d3ea6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:08.489811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a317bd6-e136-480b-b436-38c8c62d3ea6 admin admin] PATH_INFO: `/v3/projects/ed3423e6243e4316bd1e95aea4d4a531` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:08.490424 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc009368-8268-46c0-86e7-6d66f93048dd service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:08.491289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a317bd6-e136-480b-b436-38c8c62d3ea6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ed3423e6243e4316bd1e95aea4d4a531)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:08.493655 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a317bd6-e136-480b-b436-38c8c62d3ea6 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:08.497673 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe352410-ef79-40b1-9181-1b35962b592f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:08.498328 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe352410-ef79-40b1-9181-1b35962b592f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:08.498643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe352410-ef79-40b1-9181-1b35962b592f admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:08.498907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe352410-ef79-40b1-9181-1b35962b592f admin admin] PATH_INFO: `/v3/projects/b1c1956c4ef044c6b2dd7d3df2966ef4` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:08.500073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe352410-ef79-40b1-9181-1b35962b592f admin admin] RBAC: Authorizing `identity:delete_project(project_id=b1c1956c4ef044c6b2dd7d3df2966ef4)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:08.502674 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe352410-ef79-40b1-9181-1b35962b592f admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:08.510893 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:08.555367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8a317bd6-e136-480b-b436-38c8c62d3ea6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ed3423e6243e4316bd1e95aea4d4a531', 'request_id': 'req-8a317bd6-e136-480b-b436-38c8c62d3ea6'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:08.561967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8a317bd6-e136-480b-b436-38c8c62d3ea6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ed3423e6243e4316bd1e95aea4d4a531', 'request_id': 'req-8a317bd6-e136-480b-b436-38c8c62d3ea6'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:08.571191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8a317bd6-e136-480b-b436-38c8c62d3ea6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ed3423e6243e4316bd1e95aea4d4a531', 'request_id': 'req-8a317bd6-e136-480b-b436-38c8c62d3ea6'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:08.575745 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-fe352410-ef79-40b1-9181-1b35962b592f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b1c1956c4ef044c6b2dd7d3df2966ef4', 'request_id': 'req-fe352410-ef79-40b1-9181-1b35962b592f'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:08.584349 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-fe352410-ef79-40b1-9181-1b35962b592f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b1c1956c4ef044c6b2dd7d3df2966ef4', 'request_id': 'req-fe352410-ef79-40b1-9181-1b35962b592f'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:08.584920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8a317bd6-e136-480b-b436-38c8c62d3ea6 admin admin] The token cache is being invalidate because project ed3423e6243e4316bd1e95aea4d4a531 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:59:08.585221 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8a317bd6-e136-480b-b436-38c8c62d3ea6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:08.585874 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8a317bd6-e136-480b-b436-38c8c62d3ea6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:08.587593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1120/2230] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:59:08 2022] DELETE /identity/v3/projects/ed3423e6243e4316bd1e95aea4d4a531 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:08.591598 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a317bd6-e136-480b-b436-38c8c62d3ea6 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:08.592988 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-8a317bd6-e136-480b-b436-38c8c62d3ea6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:08.600171 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:08.600975 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-fe352410-ef79-40b1-9181-1b35962b592f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b1c1956c4ef044c6b2dd7d3df2966ef4', 'request_id': 'req-fe352410-ef79-40b1-9181-1b35962b592f'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:08.615664 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-fe352410-ef79-40b1-9181-1b35962b592f admin admin] The token cache is being invalidate because project b1c1956c4ef044c6b2dd7d3df2966ef4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:59:08.616098 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-fe352410-ef79-40b1-9181-1b35962b592f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:08.616865 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-fe352410-ef79-40b1-9181-1b35962b592f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:08.618521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1111/2231] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:59:08 2022] DELETE /identity/v3/projects/b1c1956c4ef044c6b2dd7d3df2966ef4 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:59:08.680831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a317bd6-e136-480b-b436-38c8c62d3ea6 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:08.766517 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61af7d99-18fb-49d9-8258-be450c4fb876 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:08.767198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61af7d99-18fb-49d9-8258-be450c4fb876 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:08.767477 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61af7d99-18fb-49d9-8258-be450c4fb876 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:08.767752 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61af7d99-18fb-49d9-8258-be450c4fb876 admin admin] PATH_INFO: `/v3/users/3ba140fc89594c2b89be8d73f9b1d8f6` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:08.769096 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61af7d99-18fb-49d9-8258-be450c4fb876 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3ba140fc89594c2b89be8d73f9b1d8f6)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:08.771882 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61af7d99-18fb-49d9-8258-be450c4fb876 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:08.827565 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-61af7d99-18fb-49d9-8258-be450c4fb876 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3ba140fc89594c2b89be8d73f9b1d8f6'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:08.836572 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-61af7d99-18fb-49d9-8258-be450c4fb876 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3ba140fc89594c2b89be8d73f9b1d8f6'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:08.850227 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-61af7d99-18fb-49d9-8258-be450c4fb876 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3ba140fc89594c2b89be8d73f9b1d8f6'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:08.850991 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-61af7d99-18fb-49d9-8258-be450c4fb876 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3ba140fc89594c2b89be8d73f9b1d8f6'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:08.857706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-61af7d99-18fb-49d9-8258-be450c4fb876 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3ba140fc89594c2b89be8d73f9b1d8f6'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:08.864201 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1121/2232] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:59:08 2022] DELETE /identity/v3/users/3ba140fc89594c2b89be8d73f9b1d8f6 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:59:09.158524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe352410-ef79-40b1-9181-1b35962b592f admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:09.160023 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-fe352410-ef79-40b1-9181-1b35962b592f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:09.172869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe352410-ef79-40b1-9181-1b35962b592f admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:09.232507 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a879f6-c8bf-4ed2-93cf-59b7cc5595da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:09.233180 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a879f6-c8bf-4ed2-93cf-59b7cc5595da admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:09.233477 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a879f6-c8bf-4ed2-93cf-59b7cc5595da admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:09.233750 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a879f6-c8bf-4ed2-93cf-59b7cc5595da admin admin] PATH_INFO: `/v3/projects/2ef37f3c7d514ccc96ad78d4ed7c4f6c` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:09.235127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a879f6-c8bf-4ed2-93cf-59b7cc5595da admin admin] RBAC: Authorizing `identity:delete_project(project_id=2ef37f3c7d514ccc96ad78d4ed7c4f6c)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:09.237018 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a879f6-c8bf-4ed2-93cf-59b7cc5595da admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:09.285456 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-74a879f6-c8bf-4ed2-93cf-59b7cc5595da admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2ef37f3c7d514ccc96ad78d4ed7c4f6c', 'request_id': 'req-74a879f6-c8bf-4ed2-93cf-59b7cc5595da'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:09.290887 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-74a879f6-c8bf-4ed2-93cf-59b7cc5595da admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2ef37f3c7d514ccc96ad78d4ed7c4f6c', 'request_id': 'req-74a879f6-c8bf-4ed2-93cf-59b7cc5595da'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:09.298621 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-74a879f6-c8bf-4ed2-93cf-59b7cc5595da admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2ef37f3c7d514ccc96ad78d4ed7c4f6c', 'request_id': 'req-74a879f6-c8bf-4ed2-93cf-59b7cc5595da'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:09.310973 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-74a879f6-c8bf-4ed2-93cf-59b7cc5595da admin admin] The token cache is being invalidate because project 2ef37f3c7d514ccc96ad78d4ed7c4f6c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:59:09.311413 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-74a879f6-c8bf-4ed2-93cf-59b7cc5595da admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:09.312095 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-74a879f6-c8bf-4ed2-93cf-59b7cc5595da admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:09.313596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1112/2233] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:59:09 2022] DELETE /identity/v3/projects/2ef37f3c7d514ccc96ad78d4ed7c4f6c => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:09.319115 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1703921-f6c5-4da6-8cfb-cbebbd57d6b2 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:09.319402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1703921-f6c5-4da6-8cfb-cbebbd57d6b2 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:09.319678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1703921-f6c5-4da6-8cfb-cbebbd57d6b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:09.339448 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-d1703921-f6c5-4da6-8cfb-cbebbd57d6b2 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:09.340641 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-d1703921-f6c5-4da6-8cfb-cbebbd57d6b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:09.369827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:09.412255 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1122/2234] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:09.420067 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edafa4cf-3dc6-45de-bf44-17172dc38014 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:09.420373 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edafa4cf-3dc6-45de-bf44-17172dc38014 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:09.420647 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edafa4cf-3dc6-45de-bf44-17172dc38014 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:09.446022 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-edafa4cf-3dc6-45de-bf44-17172dc38014 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:09.447365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-edafa4cf-3dc6-45de-bf44-17172dc38014 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:09.458334 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1113/2235] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:09.462488 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1703921-f6c5-4da6-8cfb-cbebbd57d6b2 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:09.472651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1703921-f6c5-4da6-8cfb-cbebbd57d6b2 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:09.478860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e03fff0-c150-4ea9-bb68-5d4917f6ca74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:09.479590 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e03fff0-c150-4ea9-bb68-5d4917f6ca74 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:09.479976 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e03fff0-c150-4ea9-bb68-5d4917f6ca74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:09.480354 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e03fff0-c150-4ea9-bb68-5d4917f6ca74 admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:09.481213 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e03fff0-c150-4ea9-bb68-5d4917f6ca74 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:59:09.481586 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e03fff0-c150-4ea9-bb68-5d4917f6ca74 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:09.483338 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e03fff0-c150-4ea9-bb68-5d4917f6ca74 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:09.497262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1123/2236] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:59:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 00:59:09.503177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edafa4cf-3dc6-45de-bf44-17172dc38014 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:09.521448 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edafa4cf-3dc6-45de-bf44-17172dc38014 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:09.531766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416421b5-4277-4309-afa8-29275dc9ca5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:09.532502 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416421b5-4277-4309-afa8-29275dc9ca5c admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:09.533017 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416421b5-4277-4309-afa8-29275dc9ca5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:09.533575 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416421b5-4277-4309-afa8-29275dc9ca5c admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:09.534081 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416421b5-4277-4309-afa8-29275dc9ca5c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:09.537603 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416421b5-4277-4309-afa8-29275dc9ca5c admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:09.565390 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:09.592726 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e03fff0-c150-4ea9-bb68-5d4917f6ca74 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:09.594158 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1114/2237] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:59:09 2022] POST /identity/v3/projects => generated 385 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:59:09.605159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e03fff0-c150-4ea9-bb68-5d4917f6ca74 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:09.665263 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95be2736-7ae8-4064-8512-66a794822ddb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:09.665875 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95be2736-7ae8-4064-8512-66a794822ddb admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:09.666156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95be2736-7ae8-4064-8512-66a794822ddb admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:09.666429 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95be2736-7ae8-4064-8512-66a794822ddb admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:09.667118 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95be2736-7ae8-4064-8512-66a794822ddb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:09.669110 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95be2736-7ae8-4064-8512-66a794822ddb admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:09.691904 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1124/2238] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:59:09 2022] POST /identity/v3/users => generated 293 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:59:09.696113 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416421b5-4277-4309-afa8-29275dc9ca5c admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:09.706729 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416421b5-4277-4309-afa8-29275dc9ca5c admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:09.711983 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef46db6b-31b2-40c9-97b2-76ae814b4aeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:09.712582 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef46db6b-31b2-40c9-97b2-76ae814b4aeb admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:09.712859 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef46db6b-31b2-40c9-97b2-76ae814b4aeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:09.713131 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef46db6b-31b2-40c9-97b2-76ae814b4aeb admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:09.713770 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef46db6b-31b2-40c9-97b2-76ae814b4aeb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:09.715387 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef46db6b-31b2-40c9-97b2-76ae814b4aeb admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:09.726042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1115/2239] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:09 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:59:09.730754 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95be2736-7ae8-4064-8512-66a794822ddb admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:09.741587 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95be2736-7ae8-4064-8512-66a794822ddb admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:09.746324 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d551446-b0f4-4cac-9ee4-396a0b81fd50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:09.746970 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d551446-b0f4-4cac-9ee4-396a0b81fd50 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:09.747249 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d551446-b0f4-4cac-9ee4-396a0b81fd50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:09.747524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d551446-b0f4-4cac-9ee4-396a0b81fd50 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:09.748223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d551446-b0f4-4cac-9ee4-396a0b81fd50 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:09.749927 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d551446-b0f4-4cac-9ee4-396a0b81fd50 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:09.764172 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1125/2240] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:09 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:59:09.766208 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef46db6b-31b2-40c9-97b2-76ae814b4aeb admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:09.776566 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef46db6b-31b2-40c9-97b2-76ae814b4aeb admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:09.781145 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b70127-e8ee-4180-b47f-ea068ab4b2e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:09.781786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b70127-e8ee-4180-b47f-ea068ab4b2e6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:09.782062 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b70127-e8ee-4180-b47f-ea068ab4b2e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:09.782329 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b70127-e8ee-4180-b47f-ea068ab4b2e6 admin admin] PATH_INFO: `/v3/projects/c5aa96856c14486391db1465878e7756/users/df412b495d4949df8c5cf59f0863294e/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:09.798612 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b70127-e8ee-4180-b47f-ea068ab4b2e6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c5aa96856c14486391db1465878e7756, user_id=df412b495d4949df8c5cf59f0863294e, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:09.801276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b70127-e8ee-4180-b47f-ea068ab4b2e6 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:09.815676 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1116/2241] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:59:09 2022] PUT /identity/v3/projects/c5aa96856c14486391db1465878e7756/users/df412b495d4949df8c5cf59f0863294e/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:09.828054 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec53a391-e3db-4d4f-9d6b-0f8f35688172 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:09.828336 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec53a391-e3db-4d4f-9d6b-0f8f35688172 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:09.828676 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec53a391-e3db-4d4f-9d6b-0f8f35688172 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:09.860517 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-ec53a391-e3db-4d4f-9d6b-0f8f35688172 None None] MFA Rules not processed for user `df412b495d4949df8c5cf59f0863294e`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:09.861858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-ec53a391-e3db-4d4f-9d6b-0f8f35688172 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:09.939555 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1126/2242] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:09 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:09.942021 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b70127-e8ee-4180-b47f-ea068ab4b2e6 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:10.020286 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b70127-e8ee-4180-b47f-ea068ab4b2e6 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:10.025992 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef05a94-346f-4674-8836-07a8e0250bfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:10.026864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef05a94-346f-4674-8836-07a8e0250bfc admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:10.027157 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef05a94-346f-4674-8836-07a8e0250bfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:10.027561 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef05a94-346f-4674-8836-07a8e0250bfc admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:10.028215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef05a94-346f-4674-8836-07a8e0250bfc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:10.030534 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef05a94-346f-4674-8836-07a8e0250bfc admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:10.064011 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1117/2243] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:59:09 2022] POST /identity/v3/projects => generated 383 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:59:10.065173 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec53a391-e3db-4d4f-9d6b-0f8f35688172 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:10.077128 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec53a391-e3db-4d4f-9d6b-0f8f35688172 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:10.113269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:10.166031 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9e2ecf-2725-4707-9eb4-34133677cdc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:10.166813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9e2ecf-2725-4707-9eb4-34133677cdc5 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:10.167152 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9e2ecf-2725-4707-9eb4-34133677cdc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:10.167503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9e2ecf-2725-4707-9eb4-34133677cdc5 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:10.168277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9e2ecf-2725-4707-9eb4-34133677cdc5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:10.170294 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9e2ecf-2725-4707-9eb4-34133677cdc5 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:10.193192 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1127/2244] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:59:10 2022] POST /identity/v3/users => generated 298 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:59:10.199695 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef05a94-346f-4674-8836-07a8e0250bfc admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:10.217423 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef05a94-346f-4674-8836-07a8e0250bfc admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:10.222366 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49255545-699a-4359-a2cd-d63625215eca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:10.223032 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49255545-699a-4359-a2cd-d63625215eca admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:10.223308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49255545-699a-4359-a2cd-d63625215eca admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:10.223579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49255545-699a-4359-a2cd-d63625215eca admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:10.224761 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49255545-699a-4359-a2cd-d63625215eca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:10.226342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49255545-699a-4359-a2cd-d63625215eca admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:10.237071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1118/2245] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:10 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:59:10.241975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9e2ecf-2725-4707-9eb4-34133677cdc5 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:10.252163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9e2ecf-2725-4707-9eb4-34133677cdc5 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:10.256858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d239a738-1e1d-43c8-abaa-eff97a141e84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:10.257665 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d239a738-1e1d-43c8-abaa-eff97a141e84 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:10.257945 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d239a738-1e1d-43c8-abaa-eff97a141e84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:10.258212 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d239a738-1e1d-43c8-abaa-eff97a141e84 admin admin] PATH_INFO: `/v3/projects/a50b992207134360a1b7476493e1ec86/users/3948fd55c2bc482686a3ef6747dbf74b/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:10.272652 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d239a738-1e1d-43c8-abaa-eff97a141e84 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a50b992207134360a1b7476493e1ec86, user_id=3948fd55c2bc482686a3ef6747dbf74b, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:10.275330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d239a738-1e1d-43c8-abaa-eff97a141e84 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:10.291846 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1128/2246] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:59:10 2022] PUT /identity/v3/projects/a50b992207134360a1b7476493e1ec86/users/3948fd55c2bc482686a3ef6747dbf74b/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:10.294964 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49255545-699a-4359-a2cd-d63625215eca admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:10.305615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49255545-699a-4359-a2cd-d63625215eca admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:10.359357 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d40f60-e589-4bf6-af1e-f59c6fc494e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:10.359953 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d40f60-e589-4bf6-af1e-f59c6fc494e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:10.360230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d40f60-e589-4bf6-af1e-f59c6fc494e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:10.360505 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d40f60-e589-4bf6-af1e-f59c6fc494e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:10.361133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d40f60-e589-4bf6-af1e-f59c6fc494e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:10.362771 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d40f60-e589-4bf6-af1e-f59c6fc494e1 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:10.372792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1119/2247] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:10 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:59:10.376948 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d239a738-1e1d-43c8-abaa-eff97a141e84 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:10.390078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d239a738-1e1d-43c8-abaa-eff97a141e84 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:10.394822 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c41ce20-572f-4588-a47a-fd88e52d357d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:10.395483 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c41ce20-572f-4588-a47a-fd88e52d357d admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:10.395763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c41ce20-572f-4588-a47a-fd88e52d357d admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:10.396037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c41ce20-572f-4588-a47a-fd88e52d357d admin admin] PATH_INFO: `/v3/projects/a50b992207134360a1b7476493e1ec86/users/3948fd55c2bc482686a3ef6747dbf74b/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:10.398196 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c41ce20-572f-4588-a47a-fd88e52d357d admin admin] RBAC: Authorizing `identity:create_grant(project_id=a50b992207134360a1b7476493e1ec86, user_id=3948fd55c2bc482686a3ef6747dbf74b, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:10.400866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c41ce20-572f-4588-a47a-fd88e52d357d admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:10.411922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1129/2248] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:59:10 2022] PUT /identity/v3/projects/a50b992207134360a1b7476493e1ec86/users/3948fd55c2bc482686a3ef6747dbf74b/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:10.422173 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7458c8-477d-4781-8bd9-73cd2cc56ca0 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:10.422545 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7458c8-477d-4781-8bd9-73cd2cc56ca0 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:10.422868 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7458c8-477d-4781-8bd9-73cd2cc56ca0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:10.452024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-2b7458c8-477d-4781-8bd9-73cd2cc56ca0 None None] MFA Rules not processed for user `3948fd55c2bc482686a3ef6747dbf74b`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:10.453245 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-2b7458c8-477d-4781-8bd9-73cd2cc56ca0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:10.471584 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:10.523751 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1120/2249] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:10 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:10.535362 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c41ce20-572f-4588-a47a-fd88e52d357d admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:10.537920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-2c41ce20-572f-4588-a47a-fd88e52d357d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:10.597064 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c41ce20-572f-4588-a47a-fd88e52d357d admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:10.601557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ceb51e-2cba-4d29-8d8a-b2d3e593bb06 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:10.602201 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ceb51e-2cba-4d29-8d8a-b2d3e593bb06 service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:10.602505 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ceb51e-2cba-4d29-8d8a-b2d3e593bb06 service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:10.602786 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ceb51e-2cba-4d29-8d8a-b2d3e593bb06 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:10.613323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ceb51e-2cba-4d29-8d8a-b2d3e593bb06 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:10.614913 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ceb51e-2cba-4d29-8d8a-b2d3e593bb06 service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:10.631729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1130/2250] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:59:10 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:59:10.663936 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7458c8-477d-4781-8bd9-73cd2cc56ca0 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:10.752543 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7458c8-477d-4781-8bd9-73cd2cc56ca0 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:10.757341 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdd4ca0-a791-45c5-a402-c2a96b96da80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:10.757972 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdd4ca0-a791-45c5-a402-c2a96b96da80 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:10.758432 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdd4ca0-a791-45c5-a402-c2a96b96da80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:10.758554 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdd4ca0-a791-45c5-a402-c2a96b96da80 admin admin] PATH_INFO: `/v3/users/df412b495d4949df8c5cf59f0863294e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:10.759708 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffdd4ca0-a791-45c5-a402-c2a96b96da80 admin admin] RBAC: Authorizing `identity:delete_user(user_id=df412b495d4949df8c5cf59f0863294e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:10.762218 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffdd4ca0-a791-45c5-a402-c2a96b96da80 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:10.826249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-ffdd4ca0-a791-45c5-a402-c2a96b96da80 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'df412b495d4949df8c5cf59f0863294e'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:10.841878 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-ffdd4ca0-a791-45c5-a402-c2a96b96da80 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'df412b495d4949df8c5cf59f0863294e'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:10.842644 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-ffdd4ca0-a791-45c5-a402-c2a96b96da80 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'df412b495d4949df8c5cf59f0863294e'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:10.843694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-ffdd4ca0-a791-45c5-a402-c2a96b96da80 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'df412b495d4949df8c5cf59f0863294e'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:10.852824 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-ffdd4ca0-a791-45c5-a402-c2a96b96da80 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'df412b495d4949df8c5cf59f0863294e'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:10.872798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1121/2251] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:59:10 2022] DELETE /identity/v3/users/df412b495d4949df8c5cf59f0863294e => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:10.877558 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060f0d77-b62c-4184-8722-180ccb43c2e5 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:10.877740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060f0d77-b62c-4184-8722-180ccb43c2e5 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:10.878023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060f0d77-b62c-4184-8722-180ccb43c2e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:10.897801 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-060f0d77-b62c-4184-8722-180ccb43c2e5 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:10.899041 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-060f0d77-b62c-4184-8722-180ccb43c2e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:10.961618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1131/2252] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:59:10.971357 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdd4ca0-a791-45c5-a402-c2a96b96da80 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:10.972818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ffdd4ca0-a791-45c5-a402-c2a96b96da80 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:11.023888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:11.055262 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdd4ca0-a791-45c5-a402-c2a96b96da80 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:11.059091 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8f28d0-0658-40a4-a6b5-11e64b9531d8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:11.059928 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8f28d0-0658-40a4-a6b5-11e64b9531d8 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:11.060276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8f28d0-0658-40a4-a6b5-11e64b9531d8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:11.060615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8f28d0-0658-40a4-a6b5-11e64b9531d8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:11.073382 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8f28d0-0658-40a4-a6b5-11e64b9531d8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:11.075298 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8f28d0-0658-40a4-a6b5-11e64b9531d8 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:11.087194 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1122/2253] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:59:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:59:11.374140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060f0d77-b62c-4184-8722-180ccb43c2e5 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:11.375598 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-060f0d77-b62c-4184-8722-180ccb43c2e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:11.395241 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060f0d77-b62c-4184-8722-180ccb43c2e5 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:11.400929 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd99f7e4-d270-4647-addf-f5f162ea8eb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:11.401531 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd99f7e4-d270-4647-addf-f5f162ea8eb5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:11.401830 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd99f7e4-d270-4647-addf-f5f162ea8eb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:11.402125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd99f7e4-d270-4647-addf-f5f162ea8eb5 admin admin] PATH_INFO: `/v3/projects/c5aa96856c14486391db1465878e7756` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:11.404293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd99f7e4-d270-4647-addf-f5f162ea8eb5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c5aa96856c14486391db1465878e7756)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:11.406427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd99f7e4-d270-4647-addf-f5f162ea8eb5 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:11.427511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:11.461438 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-cd99f7e4-d270-4647-addf-f5f162ea8eb5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c5aa96856c14486391db1465878e7756', 'request_id': 'req-cd99f7e4-d270-4647-addf-f5f162ea8eb5'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:11.467032 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-cd99f7e4-d270-4647-addf-f5f162ea8eb5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c5aa96856c14486391db1465878e7756', 'request_id': 'req-cd99f7e4-d270-4647-addf-f5f162ea8eb5'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:11.474323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-cd99f7e4-d270-4647-addf-f5f162ea8eb5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c5aa96856c14486391db1465878e7756', 'request_id': 'req-cd99f7e4-d270-4647-addf-f5f162ea8eb5'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:11.485361 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-cd99f7e4-d270-4647-addf-f5f162ea8eb5 admin admin] The token cache is being invalidate because project c5aa96856c14486391db1465878e7756 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:59:11.485672 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-cd99f7e4-d270-4647-addf-f5f162ea8eb5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:11.486302 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-cd99f7e4-d270-4647-addf-f5f162ea8eb5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:11.487805 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1132/2254] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:59:11 2022] DELETE /identity/v3/projects/c5aa96856c14486391db1465878e7756 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:11.492365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8f28d0-0658-40a4-a6b5-11e64b9531d8 service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:11.502250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8f28d0-0658-40a4-a6b5-11e64b9531d8 service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:11.569106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489afc9d-062b-47ba-a98c-7c59a6bf8eba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:11.569784 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489afc9d-062b-47ba-a98c-7c59a6bf8eba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:11.570147 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489afc9d-062b-47ba-a98c-7c59a6bf8eba admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:11.570444 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489afc9d-062b-47ba-a98c-7c59a6bf8eba admin admin] PATH_INFO: `/v3/users/3948fd55c2bc482686a3ef6747dbf74b` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:11.571828 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-489afc9d-062b-47ba-a98c-7c59a6bf8eba admin admin] RBAC: Authorizing `identity:delete_user(user_id=3948fd55c2bc482686a3ef6747dbf74b)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:11.574394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-489afc9d-062b-47ba-a98c-7c59a6bf8eba admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:11.630023 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-489afc9d-062b-47ba-a98c-7c59a6bf8eba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3948fd55c2bc482686a3ef6747dbf74b'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:11.636729 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-489afc9d-062b-47ba-a98c-7c59a6bf8eba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3948fd55c2bc482686a3ef6747dbf74b'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:11.637343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-489afc9d-062b-47ba-a98c-7c59a6bf8eba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3948fd55c2bc482686a3ef6747dbf74b'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:11.637985 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-489afc9d-062b-47ba-a98c-7c59a6bf8eba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3948fd55c2bc482686a3ef6747dbf74b'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:11.644891 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-489afc9d-062b-47ba-a98c-7c59a6bf8eba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3948fd55c2bc482686a3ef6747dbf74b'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:11.665011 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1123/2255] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:59:11 2022] DELETE /identity/v3/users/3948fd55c2bc482686a3ef6747dbf74b => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:11.955794 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd99f7e4-d270-4647-addf-f5f162ea8eb5 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:11.957598 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-cd99f7e4-d270-4647-addf-f5f162ea8eb5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:11.976467 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd99f7e4-d270-4647-addf-f5f162ea8eb5 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:12.048363 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533d2e41-bb71-4235-94c4-6ee24c7252dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:12.049008 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533d2e41-bb71-4235-94c4-6ee24c7252dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:12.049284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533d2e41-bb71-4235-94c4-6ee24c7252dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:12.049553 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533d2e41-bb71-4235-94c4-6ee24c7252dc admin admin] PATH_INFO: `/v3/projects/a50b992207134360a1b7476493e1ec86` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:12.050719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533d2e41-bb71-4235-94c4-6ee24c7252dc admin admin] RBAC: Authorizing `identity:delete_project(project_id=a50b992207134360a1b7476493e1ec86)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:12.053214 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533d2e41-bb71-4235-94c4-6ee24c7252dc admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:12.116861 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-533d2e41-bb71-4235-94c4-6ee24c7252dc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a50b992207134360a1b7476493e1ec86', 'request_id': 'req-533d2e41-bb71-4235-94c4-6ee24c7252dc'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:12.122659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-533d2e41-bb71-4235-94c4-6ee24c7252dc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a50b992207134360a1b7476493e1ec86', 'request_id': 'req-533d2e41-bb71-4235-94c4-6ee24c7252dc'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:12.132633 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-533d2e41-bb71-4235-94c4-6ee24c7252dc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a50b992207134360a1b7476493e1ec86', 'request_id': 'req-533d2e41-bb71-4235-94c4-6ee24c7252dc'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:12.150511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-533d2e41-bb71-4235-94c4-6ee24c7252dc admin admin] The token cache is being invalidate because project a50b992207134360a1b7476493e1ec86 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:59:12.150511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-533d2e41-bb71-4235-94c4-6ee24c7252dc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:12.150511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-533d2e41-bb71-4235-94c4-6ee24c7252dc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:12.150890 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1133/2256] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:59:11 2022] DELETE /identity/v3/projects/a50b992207134360a1b7476493e1ec86 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:12.155459 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee329b2-c8ed-4d60-b60a-8231e2f71738 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:12.155746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee329b2-c8ed-4d60-b60a-8231e2f71738 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:12.156021 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee329b2-c8ed-4d60-b60a-8231e2f71738 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:12.176304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-9ee329b2-c8ed-4d60-b60a-8231e2f71738 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:12.178113 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-9ee329b2-c8ed-4d60-b60a-8231e2f71738 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:12.232850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:12.250031 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1124/2257] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:59:12.258818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e30287-257c-4e51-bdae-c382df6a76ea None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:12.259299 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e30287-257c-4e51-bdae-c382df6a76ea None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:12.259744 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e30287-257c-4e51-bdae-c382df6a76ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:12.280978 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-67e30287-257c-4e51-bdae-c382df6a76ea None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:12.282164 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-67e30287-257c-4e51-bdae-c382df6a76ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:12.290302 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1134/2258] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:12.295015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee329b2-c8ed-4d60-b60a-8231e2f71738 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:12.305879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee329b2-c8ed-4d60-b60a-8231e2f71738 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:12.310593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0dd6aa-d39f-4508-9126-0f0d4d1f097f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:12.311339 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0dd6aa-d39f-4508-9126-0f0d4d1f097f admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:12.311619 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0dd6aa-d39f-4508-9126-0f0d4d1f097f admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:12.311892 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0dd6aa-d39f-4508-9126-0f0d4d1f097f admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:12.312572 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0dd6aa-d39f-4508-9126-0f0d4d1f097f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:59:12.312883 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0dd6aa-d39f-4508-9126-0f0d4d1f097f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:12.314494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0dd6aa-d39f-4508-9126-0f0d4d1f097f admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:12.327885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1125/2259] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:59:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:59:12.332283 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e30287-257c-4e51-bdae-c382df6a76ea None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:12.341532 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e30287-257c-4e51-bdae-c382df6a76ea None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:12.346178 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321bdff0-7aff-4336-9640-82a6f7767762 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:12.346806 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321bdff0-7aff-4336-9640-82a6f7767762 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:12.347123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321bdff0-7aff-4336-9640-82a6f7767762 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:12.347376 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321bdff0-7aff-4336-9640-82a6f7767762 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:12.348000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321bdff0-7aff-4336-9640-82a6f7767762 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:12.350740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321bdff0-7aff-4336-9640-82a6f7767762 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:12.379390 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1135/2260] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:59:12 2022] POST /identity/v3/projects => generated 393 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:59:12.383649 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0dd6aa-d39f-4508-9126-0f0d4d1f097f admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:12.394329 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0dd6aa-d39f-4508-9126-0f0d4d1f097f admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:12.463082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1775e0-6650-42ec-8377-ed734196a015 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:12.463691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1775e0-6650-42ec-8377-ed734196a015 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:12.463969 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1775e0-6650-42ec-8377-ed734196a015 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:12.464349 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1775e0-6650-42ec-8377-ed734196a015 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:12.464998 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1775e0-6650-42ec-8377-ed734196a015 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:12.466948 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1775e0-6650-42ec-8377-ed734196a015 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:12.488537 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1126/2261] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:59:12 2022] POST /identity/v3/users => generated 297 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:59:12.493383 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321bdff0-7aff-4336-9640-82a6f7767762 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:12.503694 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321bdff0-7aff-4336-9640-82a6f7767762 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:12.508491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e933c0-f6c1-4677-9c7c-a74b94e7ea07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:12.509107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e933c0-f6c1-4677-9c7c-a74b94e7ea07 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:12.509376 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e933c0-f6c1-4677-9c7c-a74b94e7ea07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:12.509645 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e933c0-f6c1-4677-9c7c-a74b94e7ea07 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:12.510339 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e933c0-f6c1-4677-9c7c-a74b94e7ea07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:12.511911 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e933c0-f6c1-4677-9c7c-a74b94e7ea07 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:12.513442 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:12.528664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1136/2262] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:12 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:59:12.529884 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1775e0-6650-42ec-8377-ed734196a015 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:12.540841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1775e0-6650-42ec-8377-ed734196a015 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:12.545714 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec77a63-fa45-4dfb-ae18-43f4a3a2eb8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:12.546313 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec77a63-fa45-4dfb-ae18-43f4a3a2eb8d admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:12.546622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec77a63-fa45-4dfb-ae18-43f4a3a2eb8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:12.546893 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec77a63-fa45-4dfb-ae18-43f4a3a2eb8d admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:12.547563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec77a63-fa45-4dfb-ae18-43f4a3a2eb8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:12.549172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec77a63-fa45-4dfb-ae18-43f4a3a2eb8d admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:12.560554 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1127/2263] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:12 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:59:12.564855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e933c0-f6c1-4677-9c7c-a74b94e7ea07 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:12.575332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e933c0-f6c1-4677-9c7c-a74b94e7ea07 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:12.579975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e298d48-abda-47cb-8493-3ca92f8ce013 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:12.580624 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e298d48-abda-47cb-8493-3ca92f8ce013 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:12.580901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e298d48-abda-47cb-8493-3ca92f8ce013 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:12.581172 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e298d48-abda-47cb-8493-3ca92f8ce013 admin admin] PATH_INFO: `/v3/projects/f7f8f3c69c10472883848cd315c825ef/users/bab6730b93a34459ac26aa47009874a1/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:12.597513 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e298d48-abda-47cb-8493-3ca92f8ce013 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7f8f3c69c10472883848cd315c825ef, user_id=bab6730b93a34459ac26aa47009874a1, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:12.600731 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e298d48-abda-47cb-8493-3ca92f8ce013 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:12.616198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1137/2264] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:59:12 2022] PUT /identity/v3/projects/f7f8f3c69c10472883848cd315c825ef/users/bab6730b93a34459ac26aa47009874a1/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:12.626392 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c4074d-8497-4b6d-b9e1-8e1ae4799067 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:12.626734 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c4074d-8497-4b6d-b9e1-8e1ae4799067 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:12.627016 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c4074d-8497-4b6d-b9e1-8e1ae4799067 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:12.658926 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-c6c4074d-8497-4b6d-b9e1-8e1ae4799067 None None] MFA Rules not processed for user `bab6730b93a34459ac26aa47009874a1`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:12.660077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c6c4074d-8497-4b6d-b9e1-8e1ae4799067 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:12.732122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1128/2265] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:12 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:12.736883 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e298d48-abda-47cb-8493-3ca92f8ce013 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:12.803263 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e298d48-abda-47cb-8493-3ca92f8ce013 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:12.809386 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a653051-6d5b-4daa-8c6e-98aa44ea2fcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:12.810097 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a653051-6d5b-4daa-8c6e-98aa44ea2fcc admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:12.810442 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a653051-6d5b-4daa-8c6e-98aa44ea2fcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:12.810737 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a653051-6d5b-4daa-8c6e-98aa44ea2fcc admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:12.811409 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a653051-6d5b-4daa-8c6e-98aa44ea2fcc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:12.813349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a653051-6d5b-4daa-8c6e-98aa44ea2fcc admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:12.841882 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1138/2266] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:59:12 2022] POST /identity/v3/projects => generated 393 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:59:12.846082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c4074d-8497-4b6d-b9e1-8e1ae4799067 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:12.857758 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c4074d-8497-4b6d-b9e1-8e1ae4799067 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:12.913605 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:12.923981 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420c5df7-5f2c-450f-bf1c-769add4db19f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:12.924605 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420c5df7-5f2c-450f-bf1c-769add4db19f admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:12.924892 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420c5df7-5f2c-450f-bf1c-769add4db19f admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:12.925167 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420c5df7-5f2c-450f-bf1c-769add4db19f admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:12.925819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420c5df7-5f2c-450f-bf1c-769add4db19f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:12.927815 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420c5df7-5f2c-450f-bf1c-769add4db19f admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:12.950296 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1129/2267] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:59:12 2022] POST /identity/v3/users => generated 303 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:59:12.954507 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a653051-6d5b-4daa-8c6e-98aa44ea2fcc admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:12.965333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a653051-6d5b-4daa-8c6e-98aa44ea2fcc admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:12.969939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d439c7aa-62a0-4229-8be2-53793168c9b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:12.970618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d439c7aa-62a0-4229-8be2-53793168c9b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:12.970919 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d439c7aa-62a0-4229-8be2-53793168c9b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:12.971187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d439c7aa-62a0-4229-8be2-53793168c9b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:12.971823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d439c7aa-62a0-4229-8be2-53793168c9b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:12.973419 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d439c7aa-62a0-4229-8be2-53793168c9b8 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:12.987013 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1139/2268] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:12 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:59:12.991142 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420c5df7-5f2c-450f-bf1c-769add4db19f admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:13.001338 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420c5df7-5f2c-450f-bf1c-769add4db19f admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:13.006274 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da27834-fe1e-4f45-a89b-31f43c11c8d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:13.006977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da27834-fe1e-4f45-a89b-31f43c11c8d6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:13.007256 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da27834-fe1e-4f45-a89b-31f43c11c8d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:13.007530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da27834-fe1e-4f45-a89b-31f43c11c8d6 admin admin] PATH_INFO: `/v3/projects/27be4c90ba3640a2bb71f4247760577f/users/a5762636f951471494c555fc4ef98fbe/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:13.025419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da27834-fe1e-4f45-a89b-31f43c11c8d6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=27be4c90ba3640a2bb71f4247760577f, user_id=a5762636f951471494c555fc4ef98fbe, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:13.028089 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da27834-fe1e-4f45-a89b-31f43c11c8d6 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:13.041736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1130/2269] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:59:12 2022] PUT /identity/v3/projects/27be4c90ba3640a2bb71f4247760577f/users/a5762636f951471494c555fc4ef98fbe/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:13.045971 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d439c7aa-62a0-4229-8be2-53793168c9b8 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:13.055847 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d439c7aa-62a0-4229-8be2-53793168c9b8 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:13.116893 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c87da338-abd0-433d-9ddf-987e5a691111 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:13.117588 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87da338-abd0-433d-9ddf-987e5a691111 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:13.117786 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87da338-abd0-433d-9ddf-987e5a691111 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:13.118056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87da338-abd0-433d-9ddf-987e5a691111 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:13.118836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c87da338-abd0-433d-9ddf-987e5a691111 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:13.120451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c87da338-abd0-433d-9ddf-987e5a691111 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:13.137332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1140/2270] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:13 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:59:13.140249 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da27834-fe1e-4f45-a89b-31f43c11c8d6 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:13.151753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da27834-fe1e-4f45-a89b-31f43c11c8d6 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:13.156687 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a41944-45d2-433f-bbf7-34f4cf08af48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:13.157437 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a41944-45d2-433f-bbf7-34f4cf08af48 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:13.157620 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a41944-45d2-433f-bbf7-34f4cf08af48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:13.158007 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a41944-45d2-433f-bbf7-34f4cf08af48 admin admin] PATH_INFO: `/v3/projects/27be4c90ba3640a2bb71f4247760577f/users/a5762636f951471494c555fc4ef98fbe/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:13.159866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a41944-45d2-433f-bbf7-34f4cf08af48 admin admin] RBAC: Authorizing `identity:create_grant(project_id=27be4c90ba3640a2bb71f4247760577f, user_id=a5762636f951471494c555fc4ef98fbe, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:13.162506 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a41944-45d2-433f-bbf7-34f4cf08af48 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:13.200777 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1131/2271] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:59:13 2022] PUT /identity/v3/projects/27be4c90ba3640a2bb71f4247760577f/users/a5762636f951471494c555fc4ef98fbe/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:13.205768 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc967cca-bbb1-435a-97d7-543cae94ed66 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:13.205768 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc967cca-bbb1-435a-97d7-543cae94ed66 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:13.205768 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc967cca-bbb1-435a-97d7-543cae94ed66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:13.250745 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-cc967cca-bbb1-435a-97d7-543cae94ed66 None None] MFA Rules not processed for user `a5762636f951471494c555fc4ef98fbe`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:13.252064 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-cc967cca-bbb1-435a-97d7-543cae94ed66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:13.290347 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:13.335055 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1141/2272] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:13 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:13.346395 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a41944-45d2-433f-bbf7-34f4cf08af48 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:13.348135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-98a41944-45d2-433f-bbf7-34f4cf08af48 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:13.406879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a41944-45d2-433f-bbf7-34f4cf08af48 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:13.410653 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d19c9ff-0e74-4c45-be72-349735bb98f5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:13.411336 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d19c9ff-0e74-4c45-be72-349735bb98f5 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:13.411556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d19c9ff-0e74-4c45-be72-349735bb98f5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:13.411827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d19c9ff-0e74-4c45-be72-349735bb98f5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:13.422368 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d19c9ff-0e74-4c45-be72-349735bb98f5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:13.423755 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d19c9ff-0e74-4c45-be72-349735bb98f5 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:13.433593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1132/2273] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:59:13 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:59:13.486835 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc967cca-bbb1-435a-97d7-543cae94ed66 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:13.488399 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-cc967cca-bbb1-435a-97d7-543cae94ed66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:13.500959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc967cca-bbb1-435a-97d7-543cae94ed66 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:13.505781 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957b020c-df67-4557-881f-20f6d9b21502 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:13.506405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957b020c-df67-4557-881f-20f6d9b21502 service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:13.506745 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957b020c-df67-4557-881f-20f6d9b21502 service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:13.507019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957b020c-df67-4557-881f-20f6d9b21502 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:13.569334 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957b020c-df67-4557-881f-20f6d9b21502 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:13.570802 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957b020c-df67-4557-881f-20f6d9b21502 service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:13.579894 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1142/2274] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:59:13 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:59:16.921678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d19c9ff-0e74-4c45-be72-349735bb98f5 service cinder] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:16.985233 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d19c9ff-0e74-4c45-be72-349735bb98f5 service cinder] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:16.989714 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaab3ae3-738a-4ae2-addd-f6ef87e35760 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:16.990586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaab3ae3-738a-4ae2-addd-f6ef87e35760 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:16.990869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaab3ae3-738a-4ae2-addd-f6ef87e35760 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:16.991154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaab3ae3-738a-4ae2-addd-f6ef87e35760 admin admin] PATH_INFO: `/v3/users/bab6730b93a34459ac26aa47009874a1` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:16.992270 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaab3ae3-738a-4ae2-addd-f6ef87e35760 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bab6730b93a34459ac26aa47009874a1)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:16.994399 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaab3ae3-738a-4ae2-addd-f6ef87e35760 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:17.047003 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-eaab3ae3-738a-4ae2-addd-f6ef87e35760 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bab6730b93a34459ac26aa47009874a1'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:17.049637 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=87,keystone:INSERT=6,keystone:DELETE=7 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:17.054962 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-eaab3ae3-738a-4ae2-addd-f6ef87e35760 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bab6730b93a34459ac26aa47009874a1'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:17.055711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-eaab3ae3-738a-4ae2-addd-f6ef87e35760 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bab6730b93a34459ac26aa47009874a1'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:17.056556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-eaab3ae3-738a-4ae2-addd-f6ef87e35760 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bab6730b93a34459ac26aa47009874a1'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:17.065299 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-eaab3ae3-738a-4ae2-addd-f6ef87e35760 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bab6730b93a34459ac26aa47009874a1'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:17.082923 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1133/2275] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:59:16 2022] DELETE /identity/v3/users/bab6730b93a34459ac26aa47009874a1 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:17.088016 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d71d7b-7d60-4405-b936-2b79d40d942e None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:17.088475 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d71d7b-7d60-4405-b936-2b79d40d942e None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:17.088590 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d71d7b-7d60-4405-b936-2b79d40d942e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:17.107532 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-33d71d7b-7d60-4405-b936-2b79d40d942e None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:17.109018 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-33d71d7b-7d60-4405-b936-2b79d40d942e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:17.169673 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1143/2276] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:59:17.179901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaab3ae3-738a-4ae2-addd-f6ef87e35760 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:17.181348 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-eaab3ae3-738a-4ae2-addd-f6ef87e35760 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:17.243177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaab3ae3-738a-4ae2-addd-f6ef87e35760 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:17.246824 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d85c8cc-a191-475a-9f31-22e39e65a1da service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:17.247434 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d85c8cc-a191-475a-9f31-22e39e65a1da service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:17.247711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d85c8cc-a191-475a-9f31-22e39e65a1da service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:17.247979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d85c8cc-a191-475a-9f31-22e39e65a1da service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:17.257809 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d85c8cc-a191-475a-9f31-22e39e65a1da service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:17.259268 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d85c8cc-a191-475a-9f31-22e39e65a1da service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:17.268931 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1134/2277] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:59:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:59:17.551373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d71d7b-7d60-4405-b936-2b79d40d942e None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:17.553470 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-33d71d7b-7d60-4405-b936-2b79d40d942e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:17.568564 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d71d7b-7d60-4405-b936-2b79d40d942e None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:17.574868 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d53a3e-baed-4966-b561-9771e95ff4ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:17.576025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d53a3e-baed-4966-b561-9771e95ff4ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:17.576377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d53a3e-baed-4966-b561-9771e95ff4ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:17.576772 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d53a3e-baed-4966-b561-9771e95ff4ef admin admin] PATH_INFO: `/v3/projects/f7f8f3c69c10472883848cd315c825ef` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:17.578318 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d53a3e-baed-4966-b561-9771e95ff4ef admin admin] RBAC: Authorizing `identity:delete_project(project_id=f7f8f3c69c10472883848cd315c825ef)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:17.580832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d53a3e-baed-4966-b561-9771e95ff4ef admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:17.632322 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-50d53a3e-baed-4966-b561-9771e95ff4ef admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f7f8f3c69c10472883848cd315c825ef', 'request_id': 'req-50d53a3e-baed-4966-b561-9771e95ff4ef'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:17.637735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-50d53a3e-baed-4966-b561-9771e95ff4ef admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f7f8f3c69c10472883848cd315c825ef', 'request_id': 'req-50d53a3e-baed-4966-b561-9771e95ff4ef'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:17.640383 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:17.648880 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-50d53a3e-baed-4966-b561-9771e95ff4ef admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f7f8f3c69c10472883848cd315c825ef', 'request_id': 'req-50d53a3e-baed-4966-b561-9771e95ff4ef'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:17.661270 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-50d53a3e-baed-4966-b561-9771e95ff4ef admin admin] The token cache is being invalidate because project f7f8f3c69c10472883848cd315c825ef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:59:17.661898 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-50d53a3e-baed-4966-b561-9771e95ff4ef admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:17.662217 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-50d53a3e-baed-4966-b561-9771e95ff4ef admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:17.663733 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1144/2278] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:59:17 2022] DELETE /identity/v3/projects/f7f8f3c69c10472883848cd315c825ef => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:17.667807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d85c8cc-a191-475a-9f31-22e39e65a1da service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:17.677405 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d85c8cc-a191-475a-9f31-22e39e65a1da service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:17.739388 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46338f1-8471-4b73-8f74-771e693b7744 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:17.740061 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46338f1-8471-4b73-8f74-771e693b7744 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:17.740337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46338f1-8471-4b73-8f74-771e693b7744 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:17.740609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46338f1-8471-4b73-8f74-771e693b7744 admin admin] PATH_INFO: `/v3/users/a5762636f951471494c555fc4ef98fbe` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:17.741807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46338f1-8471-4b73-8f74-771e693b7744 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a5762636f951471494c555fc4ef98fbe)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:17.743803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46338f1-8471-4b73-8f74-771e693b7744 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:17.794075 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-e46338f1-8471-4b73-8f74-771e693b7744 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a5762636f951471494c555fc4ef98fbe'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:17.802144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-e46338f1-8471-4b73-8f74-771e693b7744 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a5762636f951471494c555fc4ef98fbe'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:17.803255 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-e46338f1-8471-4b73-8f74-771e693b7744 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a5762636f951471494c555fc4ef98fbe'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:17.804095 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-e46338f1-8471-4b73-8f74-771e693b7744 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a5762636f951471494c555fc4ef98fbe'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:17.814993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-e46338f1-8471-4b73-8f74-771e693b7744 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a5762636f951471494c555fc4ef98fbe'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:17.823627 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:17.843110 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1135/2279] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:59:17 2022] DELETE /identity/v3/users/a5762636f951471494c555fc4ef98fbe => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:18.188715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d53a3e-baed-4966-b561-9771e95ff4ef admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:18.190277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-50d53a3e-baed-4966-b561-9771e95ff4ef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:18.208573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d53a3e-baed-4966-b561-9771e95ff4ef admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:18.280171 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb4287a-c3bb-46c6-8d5e-b0c1f48ba317 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:18.280171 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb4287a-c3bb-46c6-8d5e-b0c1f48ba317 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:18.280171 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb4287a-c3bb-46c6-8d5e-b0c1f48ba317 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:18.280171 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb4287a-c3bb-46c6-8d5e-b0c1f48ba317 admin admin] PATH_INFO: `/v3/projects/27be4c90ba3640a2bb71f4247760577f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:18.280603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb4287a-c3bb-46c6-8d5e-b0c1f48ba317 admin admin] RBAC: Authorizing `identity:delete_project(project_id=27be4c90ba3640a2bb71f4247760577f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:18.282209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb4287a-c3bb-46c6-8d5e-b0c1f48ba317 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:18.335817 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-dfb4287a-c3bb-46c6-8d5e-b0c1f48ba317 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '27be4c90ba3640a2bb71f4247760577f', 'request_id': 'req-dfb4287a-c3bb-46c6-8d5e-b0c1f48ba317'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:18.342034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-dfb4287a-c3bb-46c6-8d5e-b0c1f48ba317 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '27be4c90ba3640a2bb71f4247760577f', 'request_id': 'req-dfb4287a-c3bb-46c6-8d5e-b0c1f48ba317'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:18.352407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-dfb4287a-c3bb-46c6-8d5e-b0c1f48ba317 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '27be4c90ba3640a2bb71f4247760577f', 'request_id': 'req-dfb4287a-c3bb-46c6-8d5e-b0c1f48ba317'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:18.367150 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-dfb4287a-c3bb-46c6-8d5e-b0c1f48ba317 admin admin] The token cache is being invalidate because project 27be4c90ba3640a2bb71f4247760577f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:59:18.367459 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-dfb4287a-c3bb-46c6-8d5e-b0c1f48ba317 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:18.368172 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-dfb4287a-c3bb-46c6-8d5e-b0c1f48ba317 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:18.370224 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1145/2280] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:59:18 2022] DELETE /identity/v3/projects/27be4c90ba3640a2bb71f4247760577f => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:59:18.375162 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec275c15-e784-4a18-9a08-d02328f26074 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:18.375700 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec275c15-e784-4a18-9a08-d02328f26074 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:18.375876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec275c15-e784-4a18-9a08-d02328f26074 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:18.406483 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-ec275c15-e784-4a18-9a08-d02328f26074 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:18.407770 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ec275c15-e784-4a18-9a08-d02328f26074 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:18.479115 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1136/2281] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:59:18.487641 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240ffac4-f1a6-4b0c-b3fe-71f7ca790f12 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:18.488182 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240ffac4-f1a6-4b0c-b3fe-71f7ca790f12 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:18.488637 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240ffac4-f1a6-4b0c-b3fe-71f7ca790f12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:18.554267 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-240ffac4-f1a6-4b0c-b3fe-71f7ca790f12 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:18.555535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-240ffac4-f1a6-4b0c-b3fe-71f7ca790f12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:18.568111 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1146/2282] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:18.573175 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec275c15-e784-4a18-9a08-d02328f26074 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:18.583362 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec275c15-e784-4a18-9a08-d02328f26074 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:18.588467 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431dc695-10eb-4e2c-bc91-64d00c86d61d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:18.589017 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431dc695-10eb-4e2c-bc91-64d00c86d61d admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:18.589322 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431dc695-10eb-4e2c-bc91-64d00c86d61d admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:18.589806 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431dc695-10eb-4e2c-bc91-64d00c86d61d admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:18.590518 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431dc695-10eb-4e2c-bc91-64d00c86d61d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:59:18.590818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431dc695-10eb-4e2c-bc91-64d00c86d61d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:18.592594 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431dc695-10eb-4e2c-bc91-64d00c86d61d admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:18.604881 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1137/2283] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:59:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:59:18.609458 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240ffac4-f1a6-4b0c-b3fe-71f7ca790f12 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:18.623860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240ffac4-f1a6-4b0c-b3fe-71f7ca790f12 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:18.628637 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d65c244-4c3d-496a-b173-62188f968067 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:18.629250 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d65c244-4c3d-496a-b173-62188f968067 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:18.629546 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d65c244-4c3d-496a-b173-62188f968067 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:18.629819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d65c244-4c3d-496a-b173-62188f968067 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:18.630503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d65c244-4c3d-496a-b173-62188f968067 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:18.632941 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d65c244-4c3d-496a-b173-62188f968067 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:18.663199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1147/2284] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:59:18 2022] POST /identity/v3/projects => generated 385 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:59:18.668070 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431dc695-10eb-4e2c-bc91-64d00c86d61d admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:18.678388 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431dc695-10eb-4e2c-bc91-64d00c86d61d admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:18.741553 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6af677-75a0-4fb4-9942-0de5285f5b30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:18.742162 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6af677-75a0-4fb4-9942-0de5285f5b30 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:18.742444 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6af677-75a0-4fb4-9942-0de5285f5b30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:18.742747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6af677-75a0-4fb4-9942-0de5285f5b30 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:18.743380 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a6af677-75a0-4fb4-9942-0de5285f5b30 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:18.745297 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a6af677-75a0-4fb4-9942-0de5285f5b30 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:18.771523 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1138/2285] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:59:18 2022] POST /identity/v3/users => generated 293 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:59:18.772696 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d65c244-4c3d-496a-b173-62188f968067 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:18.783197 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d65c244-4c3d-496a-b173-62188f968067 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:18.788429 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d15463c-96c1-428d-b8c1-344f2323a348 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:18.789025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d15463c-96c1-428d-b8c1-344f2323a348 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:18.789305 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d15463c-96c1-428d-b8c1-344f2323a348 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:18.789577 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d15463c-96c1-428d-b8c1-344f2323a348 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:18.790214 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d15463c-96c1-428d-b8c1-344f2323a348 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:18.791844 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d15463c-96c1-428d-b8c1-344f2323a348 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:18.802014 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1148/2286] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:18 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:59:18.806605 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6af677-75a0-4fb4-9942-0de5285f5b30 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:18.816406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6af677-75a0-4fb4-9942-0de5285f5b30 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:18.821493 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12ace2a-7d46-4bf8-b9a7-90042e65fbff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:18.822084 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12ace2a-7d46-4bf8-b9a7-90042e65fbff admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:18.822357 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12ace2a-7d46-4bf8-b9a7-90042e65fbff admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:18.822661 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12ace2a-7d46-4bf8-b9a7-90042e65fbff admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:18.823301 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12ace2a-7d46-4bf8-b9a7-90042e65fbff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:18.825144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12ace2a-7d46-4bf8-b9a7-90042e65fbff admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:18.836216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1139/2287] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:18 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:59:18.840524 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d15463c-96c1-428d-b8c1-344f2323a348 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:18.851211 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d15463c-96c1-428d-b8c1-344f2323a348 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:18.855868 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a664685-7b25-4652-9e03-c1f7f32a8e18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:18.856519 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a664685-7b25-4652-9e03-c1f7f32a8e18 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:18.856798 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a664685-7b25-4652-9e03-c1f7f32a8e18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:18.857071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a664685-7b25-4652-9e03-c1f7f32a8e18 admin admin] PATH_INFO: `/v3/projects/9e8727c3e0a041bf9a56dd80aa8a0738/users/cb5b827fc4ba4a2095295645c6c634ac/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:18.871619 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a664685-7b25-4652-9e03-c1f7f32a8e18 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e8727c3e0a041bf9a56dd80aa8a0738, user_id=cb5b827fc4ba4a2095295645c6c634ac, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:18.874266 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a664685-7b25-4652-9e03-c1f7f32a8e18 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:18.887840 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1149/2288] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:59:18 2022] PUT /identity/v3/projects/9e8727c3e0a041bf9a56dd80aa8a0738/users/cb5b827fc4ba4a2095295645c6c634ac/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:18.898970 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f06311f-b25d-49ed-aa69-b36b2be44d63 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:18.899333 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f06311f-b25d-49ed-aa69-b36b2be44d63 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:18.899673 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f06311f-b25d-49ed-aa69-b36b2be44d63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:18.929077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-4f06311f-b25d-49ed-aa69-b36b2be44d63 None None] MFA Rules not processed for user `cb5b827fc4ba4a2095295645c6c634ac`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:18.930308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-4f06311f-b25d-49ed-aa69-b36b2be44d63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:18.979412 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:19.003710 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1140/2289] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:18 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:19.008410 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a664685-7b25-4652-9e03-c1f7f32a8e18 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:19.027811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:DELETE=8,keystone:SELECT=89,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:19.077845 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a664685-7b25-4652-9e03-c1f7f32a8e18 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:19.082551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb008a7c-a657-410b-b47a-c6b97051fb44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:19.083157 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb008a7c-a657-410b-b47a-c6b97051fb44 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:19.083435 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb008a7c-a657-410b-b47a-c6b97051fb44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:19.084258 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb008a7c-a657-410b-b47a-c6b97051fb44 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:19.084900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb008a7c-a657-410b-b47a-c6b97051fb44 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:19.086863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb008a7c-a657-410b-b47a-c6b97051fb44 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:19.116814 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1150/2290] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:59:19 2022] POST /identity/v3/projects => generated 385 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:59:19.120771 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f06311f-b25d-49ed-aa69-b36b2be44d63 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:19.131655 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f06311f-b25d-49ed-aa69-b36b2be44d63 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:19.189565 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75e0d6e-140b-4dad-a03b-5255376e6da9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:19.190402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75e0d6e-140b-4dad-a03b-5255376e6da9 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:19.190873 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75e0d6e-140b-4dad-a03b-5255376e6da9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:19.191227 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75e0d6e-140b-4dad-a03b-5255376e6da9 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:19.192112 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75e0d6e-140b-4dad-a03b-5255376e6da9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:19.194448 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75e0d6e-140b-4dad-a03b-5255376e6da9 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:19.218991 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1141/2291] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:59:19 2022] POST /identity/v3/users => generated 299 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:59:19.223383 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb008a7c-a657-410b-b47a-c6b97051fb44 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:19.235929 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb008a7c-a657-410b-b47a-c6b97051fb44 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:19.240627 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea4004c-6f7d-451c-a150-a7f61610d1b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:19.241227 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea4004c-6f7d-451c-a150-a7f61610d1b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:19.241618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea4004c-6f7d-451c-a150-a7f61610d1b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:19.241901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea4004c-6f7d-451c-a150-a7f61610d1b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:19.242579 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea4004c-6f7d-451c-a150-a7f61610d1b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:19.244196 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea4004c-6f7d-451c-a150-a7f61610d1b5 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:19.255228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1151/2292] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:19 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:59:19.259559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75e0d6e-140b-4dad-a03b-5255376e6da9 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:19.269927 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75e0d6e-140b-4dad-a03b-5255376e6da9 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:19.275324 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db76c386-59a4-40db-9dd8-303d82b5120d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:19.275982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db76c386-59a4-40db-9dd8-303d82b5120d admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:19.276261 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db76c386-59a4-40db-9dd8-303d82b5120d admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:19.276534 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db76c386-59a4-40db-9dd8-303d82b5120d admin admin] PATH_INFO: `/v3/projects/e1f209f1f6db46468ebcd5edd127734a/users/2db192e7bcb44ec699a2a39b0afd7281/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:19.291737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db76c386-59a4-40db-9dd8-303d82b5120d admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1f209f1f6db46468ebcd5edd127734a, user_id=2db192e7bcb44ec699a2a39b0afd7281, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:19.294427 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db76c386-59a4-40db-9dd8-303d82b5120d admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:19.307682 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1142/2293] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:59:19 2022] PUT /identity/v3/projects/e1f209f1f6db46468ebcd5edd127734a/users/2db192e7bcb44ec699a2a39b0afd7281/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:19.312324 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea4004c-6f7d-451c-a150-a7f61610d1b5 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:19.323025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea4004c-6f7d-451c-a150-a7f61610d1b5 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:19.392069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72dc645-59d6-48d5-8b9f-b7d8928e7c26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:19.392688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72dc645-59d6-48d5-8b9f-b7d8928e7c26 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:19.392966 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72dc645-59d6-48d5-8b9f-b7d8928e7c26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:19.393241 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72dc645-59d6-48d5-8b9f-b7d8928e7c26 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:19.393889 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72dc645-59d6-48d5-8b9f-b7d8928e7c26 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:19.395888 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72dc645-59d6-48d5-8b9f-b7d8928e7c26 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:19.409457 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1152/2294] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:19 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:59:19.410692 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db76c386-59a4-40db-9dd8-303d82b5120d admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:19.422075 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db76c386-59a4-40db-9dd8-303d82b5120d admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:19.426918 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bba721-104d-4db3-906a-a659f61aac3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:19.427579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bba721-104d-4db3-906a-a659f61aac3e admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:19.427858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bba721-104d-4db3-906a-a659f61aac3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:19.428135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bba721-104d-4db3-906a-a659f61aac3e admin admin] PATH_INFO: `/v3/projects/e1f209f1f6db46468ebcd5edd127734a/users/2db192e7bcb44ec699a2a39b0afd7281/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:19.430059 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2bba721-104d-4db3-906a-a659f61aac3e admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1f209f1f6db46468ebcd5edd127734a, user_id=2db192e7bcb44ec699a2a39b0afd7281, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:19.432722 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2bba721-104d-4db3-906a-a659f61aac3e admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:19.444537 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1143/2295] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:59:19 2022] PUT /identity/v3/projects/e1f209f1f6db46468ebcd5edd127734a/users/2db192e7bcb44ec699a2a39b0afd7281/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 00:59:19.456149 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ac7684-663d-4374-9438-77c30e066b51 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:19.456671 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ac7684-663d-4374-9438-77c30e066b51 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:19.457161 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ac7684-663d-4374-9438-77c30e066b51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:19.485469 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-45ac7684-663d-4374-9438-77c30e066b51 None None] MFA Rules not processed for user `2db192e7bcb44ec699a2a39b0afd7281`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:19.486708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-45ac7684-663d-4374-9438-77c30e066b51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:19.556322 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1153/2296] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:19 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:19.567645 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bba721-104d-4db3-906a-a659f61aac3e admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:19.570295 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c2bba721-104d-4db3-906a-a659f61aac3e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:19.651226 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bba721-104d-4db3-906a-a659f61aac3e admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:19.655548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440527f5-e134-4031-b2c2-50e6ece19bd6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:19.656165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440527f5-e134-4031-b2c2-50e6ece19bd6 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:19.656443 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440527f5-e134-4031-b2c2-50e6ece19bd6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:19.656710 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440527f5-e134-4031-b2c2-50e6ece19bd6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:19.666667 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440527f5-e134-4031-b2c2-50e6ece19bd6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:19.668110 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440527f5-e134-4031-b2c2-50e6ece19bd6 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:19.678346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1144/2297] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:59:19 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:59:20.351065 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ac7684-663d-4374-9438-77c30e066b51 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:20.363823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:20.418666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ac7684-663d-4374-9438-77c30e066b51 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:20.423439 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3ae0ff-9c0d-4c9f-b0b0-cff960b19360 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:20.424069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3ae0ff-9c0d-4c9f-b0b0-cff960b19360 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:20.424364 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3ae0ff-9c0d-4c9f-b0b0-cff960b19360 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:20.424622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3ae0ff-9c0d-4c9f-b0b0-cff960b19360 admin admin] PATH_INFO: `/v3/users/cb5b827fc4ba4a2095295645c6c634ac` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:20.425798 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3ae0ff-9c0d-4c9f-b0b0-cff960b19360 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cb5b827fc4ba4a2095295645c6c634ac)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:20.428278 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3ae0ff-9c0d-4c9f-b0b0-cff960b19360 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:20.475927 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2f3ae0ff-9c0d-4c9f-b0b0-cff960b19360 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cb5b827fc4ba4a2095295645c6c634ac'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:20.484889 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2f3ae0ff-9c0d-4c9f-b0b0-cff960b19360 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cb5b827fc4ba4a2095295645c6c634ac'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:20.500919 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2f3ae0ff-9c0d-4c9f-b0b0-cff960b19360 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cb5b827fc4ba4a2095295645c6c634ac'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:20.501753 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2f3ae0ff-9c0d-4c9f-b0b0-cff960b19360 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cb5b827fc4ba4a2095295645c6c634ac'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:20.513991 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2f3ae0ff-9c0d-4c9f-b0b0-cff960b19360 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cb5b827fc4ba4a2095295645c6c634ac'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:20.520375 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1154/2298] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:59:20 2022] DELETE /identity/v3/users/cb5b827fc4ba4a2095295645c6c634ac => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:20.526339 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9462f74-d5bf-417d-b4cd-950c7843e0d1 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:20.526937 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9462f74-d5bf-417d-b4cd-950c7843e0d1 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:20.526937 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9462f74-d5bf-417d-b4cd-950c7843e0d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:20.551994 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-e9462f74-d5bf-417d-b4cd-950c7843e0d1 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:20.554179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-e9462f74-d5bf-417d-b4cd-950c7843e0d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:20.601876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:20.637988 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1145/2299] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:20.655638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3ae0ff-9c0d-4c9f-b0b0-cff960b19360 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:20.655638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-2f3ae0ff-9c0d-4c9f-b0b0-cff960b19360 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:20.711580 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3ae0ff-9c0d-4c9f-b0b0-cff960b19360 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:20.715428 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6f773d-7029-4644-ab5b-2a16099a53a0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:20.716036 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6f773d-7029-4644-ab5b-2a16099a53a0 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:20.716307 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6f773d-7029-4644-ab5b-2a16099a53a0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:20.716571 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6f773d-7029-4644-ab5b-2a16099a53a0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:20.726043 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6f773d-7029-4644-ab5b-2a16099a53a0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:20.727526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6f773d-7029-4644-ab5b-2a16099a53a0 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:20.736732 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1155/2300] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:59:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:59:21.016877 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9462f74-d5bf-417d-b4cd-950c7843e0d1 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:21.019164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-e9462f74-d5bf-417d-b4cd-950c7843e0d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:21.035627 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9462f74-d5bf-417d-b4cd-950c7843e0d1 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:21.041183 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3f088a-b7eb-4345-a88f-05a7b3aeb712 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:21.042199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3f088a-b7eb-4345-a88f-05a7b3aeb712 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:21.042751 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3f088a-b7eb-4345-a88f-05a7b3aeb712 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:21.043154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3f088a-b7eb-4345-a88f-05a7b3aeb712 admin admin] PATH_INFO: `/v3/projects/9e8727c3e0a041bf9a56dd80aa8a0738` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:21.045101 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3f088a-b7eb-4345-a88f-05a7b3aeb712 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9e8727c3e0a041bf9a56dd80aa8a0738)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:21.047735 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3f088a-b7eb-4345-a88f-05a7b3aeb712 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:21.096701 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-2d3f088a-b7eb-4345-a88f-05a7b3aeb712 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9e8727c3e0a041bf9a56dd80aa8a0738', 'request_id': 'req-2d3f088a-b7eb-4345-a88f-05a7b3aeb712'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:21.102313 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-2d3f088a-b7eb-4345-a88f-05a7b3aeb712 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9e8727c3e0a041bf9a56dd80aa8a0738', 'request_id': 'req-2d3f088a-b7eb-4345-a88f-05a7b3aeb712'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:21.110838 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-2d3f088a-b7eb-4345-a88f-05a7b3aeb712 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9e8727c3e0a041bf9a56dd80aa8a0738', 'request_id': 'req-2d3f088a-b7eb-4345-a88f-05a7b3aeb712'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:21.123150 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-2d3f088a-b7eb-4345-a88f-05a7b3aeb712 admin admin] The token cache is being invalidate because project 9e8727c3e0a041bf9a56dd80aa8a0738 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:59:21.123439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-2d3f088a-b7eb-4345-a88f-05a7b3aeb712 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:21.124178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-2d3f088a-b7eb-4345-a88f-05a7b3aeb712 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:21.125757 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1146/2301] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:59:21 2022] DELETE /identity/v3/projects/9e8727c3e0a041bf9a56dd80aa8a0738 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:21.130180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6f773d-7029-4644-ab5b-2a16099a53a0 service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:21.140829 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6f773d-7029-4644-ab5b-2a16099a53a0 service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:21.171742 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:21.204744 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92787479-1b73-4450-8788-fde24c1dcb5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:21.205456 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92787479-1b73-4450-8788-fde24c1dcb5f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:21.205740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92787479-1b73-4450-8788-fde24c1dcb5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:21.206015 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92787479-1b73-4450-8788-fde24c1dcb5f admin admin] PATH_INFO: `/v3/users/2db192e7bcb44ec699a2a39b0afd7281` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:21.207175 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92787479-1b73-4450-8788-fde24c1dcb5f admin admin] RBAC: Authorizing `identity:delete_user(user_id=2db192e7bcb44ec699a2a39b0afd7281)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:21.209767 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92787479-1b73-4450-8788-fde24c1dcb5f admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:21.270504 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-92787479-1b73-4450-8788-fde24c1dcb5f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2db192e7bcb44ec699a2a39b0afd7281'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:21.283948 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-92787479-1b73-4450-8788-fde24c1dcb5f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2db192e7bcb44ec699a2a39b0afd7281'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:21.296723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-92787479-1b73-4450-8788-fde24c1dcb5f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2db192e7bcb44ec699a2a39b0afd7281'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:21.297566 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-92787479-1b73-4450-8788-fde24c1dcb5f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2db192e7bcb44ec699a2a39b0afd7281'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:21.304684 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-92787479-1b73-4450-8788-fde24c1dcb5f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2db192e7bcb44ec699a2a39b0afd7281'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:21.311161 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1156/2302] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:59:21 2022] DELETE /identity/v3/users/2db192e7bcb44ec699a2a39b0afd7281 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:21.656021 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3f088a-b7eb-4345-a88f-05a7b3aeb712 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:21.659871 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-2d3f088a-b7eb-4345-a88f-05a7b3aeb712 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:21.672197 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3f088a-b7eb-4345-a88f-05a7b3aeb712 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:21.769588 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1b4454-6fe6-4cbb-bc81-3d2a5528f56a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:21.770876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1b4454-6fe6-4cbb-bc81-3d2a5528f56a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:21.771204 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1b4454-6fe6-4cbb-bc81-3d2a5528f56a admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:21.771473 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1b4454-6fe6-4cbb-bc81-3d2a5528f56a admin admin] PATH_INFO: `/v3/projects/e1f209f1f6db46468ebcd5edd127734a` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:21.772624 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1b4454-6fe6-4cbb-bc81-3d2a5528f56a admin admin] RBAC: Authorizing `identity:delete_project(project_id=e1f209f1f6db46468ebcd5edd127734a)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:21.774606 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1b4454-6fe6-4cbb-bc81-3d2a5528f56a admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:21.831885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-2b1b4454-6fe6-4cbb-bc81-3d2a5528f56a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e1f209f1f6db46468ebcd5edd127734a', 'request_id': 'req-2b1b4454-6fe6-4cbb-bc81-3d2a5528f56a'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:21.837948 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-2b1b4454-6fe6-4cbb-bc81-3d2a5528f56a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e1f209f1f6db46468ebcd5edd127734a', 'request_id': 'req-2b1b4454-6fe6-4cbb-bc81-3d2a5528f56a'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:21.845778 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-2b1b4454-6fe6-4cbb-bc81-3d2a5528f56a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e1f209f1f6db46468ebcd5edd127734a', 'request_id': 'req-2b1b4454-6fe6-4cbb-bc81-3d2a5528f56a'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:21.847428 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:21.862144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-2b1b4454-6fe6-4cbb-bc81-3d2a5528f56a admin admin] The token cache is being invalidate because project e1f209f1f6db46468ebcd5edd127734a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:59:21.862433 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-2b1b4454-6fe6-4cbb-bc81-3d2a5528f56a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:21.863117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-2b1b4454-6fe6-4cbb-bc81-3d2a5528f56a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:21.864668 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1147/2303] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:59:21 2022] DELETE /identity/v3/projects/e1f209f1f6db46468ebcd5edd127734a => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:21.870126 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4b3754-ba6d-4d3c-bb03-8152a36f9a69 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:21.870644 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4b3754-ba6d-4d3c-bb03-8152a36f9a69 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:21.870739 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4b3754-ba6d-4d3c-bb03-8152a36f9a69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:21.891639 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-0a4b3754-ba6d-4d3c-bb03-8152a36f9a69 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:21.892972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-0a4b3754-ba6d-4d3c-bb03-8152a36f9a69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:21.973175 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1157/2304] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:21.975590 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747540cc-c97b-477f-814d-8c010e1bb1c5 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:21.975931 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747540cc-c97b-477f-814d-8c010e1bb1c5 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:21.976204 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747540cc-c97b-477f-814d-8c010e1bb1c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:21.997127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-747540cc-c97b-477f-814d-8c010e1bb1c5 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:21.998340 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-747540cc-c97b-477f-814d-8c010e1bb1c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:22.006534 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1148/2305] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:22.011262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4b3754-ba6d-4d3c-bb03-8152a36f9a69 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:22.022268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4b3754-ba6d-4d3c-bb03-8152a36f9a69 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:22.028818 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa848bf9-f822-45f4-acab-39af301b12c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:22.029610 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa848bf9-f822-45f4-acab-39af301b12c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:22.030207 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa848bf9-f822-45f4-acab-39af301b12c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:22.030447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa848bf9-f822-45f4-acab-39af301b12c7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:22.031143 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa848bf9-f822-45f4-acab-39af301b12c7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:59:22.031444 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa848bf9-f822-45f4-acab-39af301b12c7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:22.033062 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa848bf9-f822-45f4-acab-39af301b12c7 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:22.054265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1158/2306] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:59:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:59:22.055415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747540cc-c97b-477f-814d-8c010e1bb1c5 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:22.064745 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747540cc-c97b-477f-814d-8c010e1bb1c5 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:22.070275 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8469f07-5f8b-4bff-aa7a-9e21140d8dfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:22.070925 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8469f07-5f8b-4bff-aa7a-9e21140d8dfc admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:22.071196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8469f07-5f8b-4bff-aa7a-9e21140d8dfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:22.071471 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8469f07-5f8b-4bff-aa7a-9e21140d8dfc admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:22.074627 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8469f07-5f8b-4bff-aa7a-9e21140d8dfc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:22.074627 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8469f07-5f8b-4bff-aa7a-9e21140d8dfc admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:22.108224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1149/2307] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:59:22 2022] POST /identity/v3/projects => generated 385 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:59:22.110285 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa848bf9-f822-45f4-acab-39af301b12c7 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:22.126444 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa848bf9-f822-45f4-acab-39af301b12c7 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:22.173208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:22.196162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51847029-ff0c-4fb2-8663-5dc977f0c974 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:22.196772 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51847029-ff0c-4fb2-8663-5dc977f0c974 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:22.197049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51847029-ff0c-4fb2-8663-5dc977f0c974 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:22.197316 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51847029-ff0c-4fb2-8663-5dc977f0c974 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:22.197975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51847029-ff0c-4fb2-8663-5dc977f0c974 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:22.200179 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51847029-ff0c-4fb2-8663-5dc977f0c974 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:22.231135 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1159/2308] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:59:22 2022] POST /identity/v3/users => generated 293 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:59:22.232544 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8469f07-5f8b-4bff-aa7a-9e21140d8dfc admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:22.244183 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8469f07-5f8b-4bff-aa7a-9e21140d8dfc admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:22.249236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbb5a18-0eea-456d-a709-e54825ea4cf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:22.249951 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbb5a18-0eea-456d-a709-e54825ea4cf9 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:22.250224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbb5a18-0eea-456d-a709-e54825ea4cf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:22.250518 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbb5a18-0eea-456d-a709-e54825ea4cf9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:22.251164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fbb5a18-0eea-456d-a709-e54825ea4cf9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:22.252776 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fbb5a18-0eea-456d-a709-e54825ea4cf9 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:22.263146 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1150/2309] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:22 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:59:22.267359 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51847029-ff0c-4fb2-8663-5dc977f0c974 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:22.290206 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51847029-ff0c-4fb2-8663-5dc977f0c974 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:22.296418 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7717cf76-4c6a-49b7-b9e3-7b8772b36566 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:22.297231 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7717cf76-4c6a-49b7-b9e3-7b8772b36566 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:22.297512 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7717cf76-4c6a-49b7-b9e3-7b8772b36566 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:22.297785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7717cf76-4c6a-49b7-b9e3-7b8772b36566 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:22.298518 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7717cf76-4c6a-49b7-b9e3-7b8772b36566 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:22.300132 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7717cf76-4c6a-49b7-b9e3-7b8772b36566 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:22.312756 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1160/2310] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:22 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:59:22.316636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbb5a18-0eea-456d-a709-e54825ea4cf9 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:22.326788 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbb5a18-0eea-456d-a709-e54825ea4cf9 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:22.331681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe461c7-fbdd-40b0-b82f-867cc1ff9016 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:22.332339 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe461c7-fbdd-40b0-b82f-867cc1ff9016 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:22.332640 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe461c7-fbdd-40b0-b82f-867cc1ff9016 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:22.332899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe461c7-fbdd-40b0-b82f-867cc1ff9016 admin admin] PATH_INFO: `/v3/projects/071b63e294fe42b5a26c6eaeecbd204d/users/12d50655e6064095a6dbc7350d6cc30f/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:22.349745 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe461c7-fbdd-40b0-b82f-867cc1ff9016 admin admin] RBAC: Authorizing `identity:create_grant(project_id=071b63e294fe42b5a26c6eaeecbd204d, user_id=12d50655e6064095a6dbc7350d6cc30f, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:22.352479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe461c7-fbdd-40b0-b82f-867cc1ff9016 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:22.368441 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1151/2311] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:59:22 2022] PUT /identity/v3/projects/071b63e294fe42b5a26c6eaeecbd204d/users/12d50655e6064095a6dbc7350d6cc30f/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:22.380666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b0a977-d243-43f1-b8bf-638600559231 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:22.380960 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b0a977-d243-43f1-b8bf-638600559231 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:22.381242 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b0a977-d243-43f1-b8bf-638600559231 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:22.411269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-c8b0a977-d243-43f1-b8bf-638600559231 None None] MFA Rules not processed for user `12d50655e6064095a6dbc7350d6cc30f`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:22.413371 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c8b0a977-d243-43f1-b8bf-638600559231 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:22.488211 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1161/2312] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:22 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:22.493188 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe461c7-fbdd-40b0-b82f-867cc1ff9016 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:22.570982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe461c7-fbdd-40b0-b82f-867cc1ff9016 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:22.576471 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6df369a-33ca-44e5-83ac-9aee808ed66f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:22.577085 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6df369a-33ca-44e5-83ac-9aee808ed66f admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:22.577357 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6df369a-33ca-44e5-83ac-9aee808ed66f admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:22.577625 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6df369a-33ca-44e5-83ac-9aee808ed66f admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:22.578283 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6df369a-33ca-44e5-83ac-9aee808ed66f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:22.580188 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6df369a-33ca-44e5-83ac-9aee808ed66f admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:22.610000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1152/2313] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:59:22 2022] POST /identity/v3/projects => generated 383 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:59:22.614170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b0a977-d243-43f1-b8bf-638600559231 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:22.627267 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b0a977-d243-43f1-b8bf-638600559231 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:22.694703 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a6488d-ab78-451a-b0fa-3dd0c3cea8b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:22.695333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a6488d-ab78-451a-b0fa-3dd0c3cea8b7 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:22.695617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a6488d-ab78-451a-b0fa-3dd0c3cea8b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:22.695906 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a6488d-ab78-451a-b0fa-3dd0c3cea8b7 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:22.696551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a6488d-ab78-451a-b0fa-3dd0c3cea8b7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:22.698527 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a6488d-ab78-451a-b0fa-3dd0c3cea8b7 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:22.736248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1162/2314] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:59:22 2022] POST /identity/v3/users => generated 292 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:59:22.742177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6df369a-33ca-44e5-83ac-9aee808ed66f admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:22.755227 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6df369a-33ca-44e5-83ac-9aee808ed66f admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:22.760833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd66d253-22bd-4986-8878-c8b09e3c7230 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:22.761772 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd66d253-22bd-4986-8878-c8b09e3c7230 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:22.761772 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd66d253-22bd-4986-8878-c8b09e3c7230 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:22.762071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd66d253-22bd-4986-8878-c8b09e3c7230 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:22.762935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd66d253-22bd-4986-8878-c8b09e3c7230 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:22.764560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd66d253-22bd-4986-8878-c8b09e3c7230 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:22.795872 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1153/2315] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:22 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:59:22.797225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a6488d-ab78-451a-b0fa-3dd0c3cea8b7 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:22.814800 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a6488d-ab78-451a-b0fa-3dd0c3cea8b7 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:22.820613 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5d5c6c-170c-4bb1-857f-a80c59d42629 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:22.821257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5d5c6c-170c-4bb1-857f-a80c59d42629 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:22.821537 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5d5c6c-170c-4bb1-857f-a80c59d42629 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:22.821811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5d5c6c-170c-4bb1-857f-a80c59d42629 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:22.822516 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5d5c6c-170c-4bb1-857f-a80c59d42629 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:22.824229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5d5c6c-170c-4bb1-857f-a80c59d42629 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:22.835531 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1163/2316] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:22 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:59:22.839654 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd66d253-22bd-4986-8878-c8b09e3c7230 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:22.850293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd66d253-22bd-4986-8878-c8b09e3c7230 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:22.855185 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab20052-8a74-499d-8935-7fce669de82f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:22.855858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab20052-8a74-499d-8935-7fce669de82f admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:22.856137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab20052-8a74-499d-8935-7fce669de82f admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:22.856413 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab20052-8a74-499d-8935-7fce669de82f admin admin] PATH_INFO: `/v3/projects/2cde2b94cae2420f96f0c1121b815181/users/872749a884f140079a35d42a979d539d/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:22.873431 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab20052-8a74-499d-8935-7fce669de82f admin admin] RBAC: Authorizing `identity:create_grant(project_id=2cde2b94cae2420f96f0c1121b815181, user_id=872749a884f140079a35d42a979d539d, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:22.876945 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab20052-8a74-499d-8935-7fce669de82f admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:22.894117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1154/2317] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:59:22 2022] PUT /identity/v3/projects/2cde2b94cae2420f96f0c1121b815181/users/872749a884f140079a35d42a979d539d/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:22.905632 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da1d1c1-356e-47bf-870a-4847f895356d None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:22.906182 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da1d1c1-356e-47bf-870a-4847f895356d None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:22.906182 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da1d1c1-356e-47bf-870a-4847f895356d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:22.936810 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-8da1d1c1-356e-47bf-870a-4847f895356d None None] MFA Rules not processed for user `872749a884f140079a35d42a979d539d`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:22.938042 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-8da1d1c1-356e-47bf-870a-4847f895356d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:22.951682 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:23.015461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1164/2318] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:22 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:23.021455 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab20052-8a74-499d-8935-7fce669de82f admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:23.068690 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:23.091570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab20052-8a74-499d-8935-7fce669de82f admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:23.096731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37974299-a49b-456f-95d4-4d1b1bee7bd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:23.097380 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37974299-a49b-456f-95d4-4d1b1bee7bd3 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:23.097658 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37974299-a49b-456f-95d4-4d1b1bee7bd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:23.097928 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37974299-a49b-456f-95d4-4d1b1bee7bd3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:23.098616 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37974299-a49b-456f-95d4-4d1b1bee7bd3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:23.100561 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37974299-a49b-456f-95d4-4d1b1bee7bd3 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:23.130384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1155/2319] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:59:23 2022] POST /identity/v3/projects => generated 385 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:59:23.135414 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da1d1c1-356e-47bf-870a-4847f895356d None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:23.147161 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da1d1c1-356e-47bf-870a-4847f895356d None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:23.214897 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83f17c8-5cea-45d4-ae43-b825c0608111 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:23.215685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83f17c8-5cea-45d4-ae43-b825c0608111 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:23.216049 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83f17c8-5cea-45d4-ae43-b825c0608111 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:23.216469 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83f17c8-5cea-45d4-ae43-b825c0608111 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:23.217213 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83f17c8-5cea-45d4-ae43-b825c0608111 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:23.219363 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83f17c8-5cea-45d4-ae43-b825c0608111 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:23.243157 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1165/2320] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:59:23 2022] POST /identity/v3/users => generated 299 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 00:59:23.247634 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37974299-a49b-456f-95d4-4d1b1bee7bd3 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:23.259698 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37974299-a49b-456f-95d4-4d1b1bee7bd3 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:23.265683 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43b847b-2de0-4d52-9e2e-4e8a62506a46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:23.266257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43b847b-2de0-4d52-9e2e-4e8a62506a46 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:23.266574 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43b847b-2de0-4d52-9e2e-4e8a62506a46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:23.266840 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43b847b-2de0-4d52-9e2e-4e8a62506a46 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:23.267486 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43b847b-2de0-4d52-9e2e-4e8a62506a46 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:23.269079 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43b847b-2de0-4d52-9e2e-4e8a62506a46 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:23.281088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1156/2321] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:23 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:59:23.285931 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83f17c8-5cea-45d4-ae43-b825c0608111 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:23.298904 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83f17c8-5cea-45d4-ae43-b825c0608111 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:23.304539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c065509b-328d-48fe-929d-e6adfd98f3e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:23.305333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c065509b-328d-48fe-929d-e6adfd98f3e5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:23.305590 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c065509b-328d-48fe-929d-e6adfd98f3e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:23.305866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c065509b-328d-48fe-929d-e6adfd98f3e5 admin admin] PATH_INFO: `/v3/projects/a2d7fb49579646f195f993081ce7ea06/users/5ba9ef3d01b34035be12c373dbe76b5e/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:23.325186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c065509b-328d-48fe-929d-e6adfd98f3e5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a2d7fb49579646f195f993081ce7ea06, user_id=5ba9ef3d01b34035be12c373dbe76b5e, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:23.327918 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c065509b-328d-48fe-929d-e6adfd98f3e5 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:23.343198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1166/2322] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:59:23 2022] PUT /identity/v3/projects/a2d7fb49579646f195f993081ce7ea06/users/5ba9ef3d01b34035be12c373dbe76b5e/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:23.347826 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43b847b-2de0-4d52-9e2e-4e8a62506a46 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:23.358628 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43b847b-2de0-4d52-9e2e-4e8a62506a46 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:23.430025 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb71cf3-8f05-42cb-8c5b-e393b3b2c279 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:23.430671 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb71cf3-8f05-42cb-8c5b-e393b3b2c279 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:23.430954 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb71cf3-8f05-42cb-8c5b-e393b3b2c279 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:23.431228 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb71cf3-8f05-42cb-8c5b-e393b3b2c279 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:23.431878 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb71cf3-8f05-42cb-8c5b-e393b3b2c279 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:23.433470 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb71cf3-8f05-42cb-8c5b-e393b3b2c279 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:23.444834 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1157/2323] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:23 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:59:23.449659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c065509b-328d-48fe-929d-e6adfd98f3e5 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:23.462919 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c065509b-328d-48fe-929d-e6adfd98f3e5 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:23.467744 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7bab5a-831d-441f-acff-8aa2bb8f1691 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:23.468405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7bab5a-831d-441f-acff-8aa2bb8f1691 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:23.468679 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7bab5a-831d-441f-acff-8aa2bb8f1691 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:23.468950 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7bab5a-831d-441f-acff-8aa2bb8f1691 admin admin] PATH_INFO: `/v3/projects/a2d7fb49579646f195f993081ce7ea06/users/5ba9ef3d01b34035be12c373dbe76b5e/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:23.470865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7bab5a-831d-441f-acff-8aa2bb8f1691 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a2d7fb49579646f195f993081ce7ea06, user_id=5ba9ef3d01b34035be12c373dbe76b5e, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:23.473480 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7bab5a-831d-441f-acff-8aa2bb8f1691 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:23.484055 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1167/2324] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:59:23 2022] PUT /identity/v3/projects/a2d7fb49579646f195f993081ce7ea06/users/5ba9ef3d01b34035be12c373dbe76b5e/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:23.494906 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac4b39b-eaeb-4f6f-96f9-33a5c906e442 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:23.495190 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac4b39b-eaeb-4f6f-96f9-33a5c906e442 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:23.495467 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac4b39b-eaeb-4f6f-96f9-33a5c906e442 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:23.524943 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-1ac4b39b-eaeb-4f6f-96f9-33a5c906e442 None None] MFA Rules not processed for user `5ba9ef3d01b34035be12c373dbe76b5e`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:23.526230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-1ac4b39b-eaeb-4f6f-96f9-33a5c906e442 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:23.594738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1158/2325] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:23 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:59:23.606050 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7bab5a-831d-441f-acff-8aa2bb8f1691 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:23.607797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-9a7bab5a-831d-441f-acff-8aa2bb8f1691 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:23.667212 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7bab5a-831d-441f-acff-8aa2bb8f1691 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:23.670948 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c647ad43-183c-4dd0-9586-df5385b3623e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:23.671560 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c647ad43-183c-4dd0-9586-df5385b3623e service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:23.671837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c647ad43-183c-4dd0-9586-df5385b3623e service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:23.672109 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c647ad43-183c-4dd0-9586-df5385b3623e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:23.692605 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:23.730563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c647ad43-183c-4dd0-9586-df5385b3623e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:23.732130 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c647ad43-183c-4dd0-9586-df5385b3623e service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:23.742206 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1168/2326] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:59:23 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:59:25.295353 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac4b39b-eaeb-4f6f-96f9-33a5c906e442 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:25.297135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-1ac4b39b-eaeb-4f6f-96f9-33a5c906e442 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:25.310676 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac4b39b-eaeb-4f6f-96f9-33a5c906e442 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:25.314297 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ac7e48-574f-4fbf-8f0d-f807b7c21c2f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:25.314944 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ac7e48-574f-4fbf-8f0d-f807b7c21c2f service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:25.315220 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ac7e48-574f-4fbf-8f0d-f807b7c21c2f service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:25.315485 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ac7e48-574f-4fbf-8f0d-f807b7c21c2f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:25.350770 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:25.375905 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ac7e48-574f-4fbf-8f0d-f807b7c21c2f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:25.377337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ac7e48-574f-4fbf-8f0d-f807b7c21c2f service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:25.386521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1159/2327] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:59:25 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:59:28.017747 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c647ad43-183c-4dd0-9586-df5385b3623e service cinder] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:28.029895 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c647ad43-183c-4dd0-9586-df5385b3623e service cinder] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:28.035670 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786505c6-ecc0-4dbc-a818-cc38c75ffdcc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:28.038152 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786505c6-ecc0-4dbc-a818-cc38c75ffdcc service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:28.038974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786505c6-ecc0-4dbc-a818-cc38c75ffdcc service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:28.039419 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786505c6-ecc0-4dbc-a818-cc38c75ffdcc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:28.051694 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786505c6-ecc0-4dbc-a818-cc38c75ffdcc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:28.053355 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786505c6-ecc0-4dbc-a818-cc38c75ffdcc service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:28.068501 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1169/2328] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:59:28 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:59:31.986262 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edc8eaa-3f28-4bf6-8e48-32f1527bdfaf None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:31.986847 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edc8eaa-3f28-4bf6-8e48-32f1527bdfaf None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:31.986943 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edc8eaa-3f28-4bf6-8e48-32f1527bdfaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:32.008009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-6edc8eaa-3f28-4bf6-8e48-32f1527bdfaf None None] MFA Rules not processed for user `54d4e437036640b58fda503c4d213f5c`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:32.009524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-6edc8eaa-3f28-4bf6-8e48-32f1527bdfaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:32.083611 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1160/2329] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:31 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:37.478023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786505c6-ecc0-4dbc-a818-cc38c75ffdcc service cinder] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:37.547161 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786505c6-ecc0-4dbc-a818-cc38c75ffdcc service cinder] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:37.551979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bc539a-1cfb-400b-b5b0-cf68c1860655 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:37.552659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bc539a-1cfb-400b-b5b0-cf68c1860655 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:37.552938 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bc539a-1cfb-400b-b5b0-cf68c1860655 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:37.553211 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bc539a-1cfb-400b-b5b0-cf68c1860655 admin admin] PATH_INFO: `/v3/users/12d50655e6064095a6dbc7350d6cc30f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:37.554914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2bc539a-1cfb-400b-b5b0-cf68c1860655 admin admin] RBAC: Authorizing `identity:delete_user(user_id=12d50655e6064095a6dbc7350d6cc30f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:37.556867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2bc539a-1cfb-400b-b5b0-cf68c1860655 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:37.613569 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-e2bc539a-1cfb-400b-b5b0-cf68c1860655 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '12d50655e6064095a6dbc7350d6cc30f'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:37.622423 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-e2bc539a-1cfb-400b-b5b0-cf68c1860655 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '12d50655e6064095a6dbc7350d6cc30f'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:37.635619 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-e2bc539a-1cfb-400b-b5b0-cf68c1860655 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '12d50655e6064095a6dbc7350d6cc30f'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:37.636305 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-e2bc539a-1cfb-400b-b5b0-cf68c1860655 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '12d50655e6064095a6dbc7350d6cc30f'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:37.644432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-e2bc539a-1cfb-400b-b5b0-cf68c1860655 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '12d50655e6064095a6dbc7350d6cc30f'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:37.651438 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1170/2330] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:59:37 2022] DELETE /identity/v3/users/12d50655e6064095a6dbc7350d6cc30f => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:37.656015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c919f1fe-ef7d-4c17-8b23-78a0aefa655b None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:37.656484 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c919f1fe-ef7d-4c17-8b23-78a0aefa655b None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:37.656582 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c919f1fe-ef7d-4c17-8b23-78a0aefa655b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:37.676646 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-c919f1fe-ef7d-4c17-8b23-78a0aefa655b None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:37.677879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c919f1fe-ef7d-4c17-8b23-78a0aefa655b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:37.751425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1161/2331] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:37.764300 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bc539a-1cfb-400b-b5b0-cf68c1860655 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:37.765862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-e2bc539a-1cfb-400b-b5b0-cf68c1860655 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:37.816396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:37.847335 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bc539a-1cfb-400b-b5b0-cf68c1860655 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:37.852020 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6043d843-e92f-4964-97cf-5ab54b97a651 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:37.852683 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6043d843-e92f-4964-97cf-5ab54b97a651 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:37.853031 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6043d843-e92f-4964-97cf-5ab54b97a651 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:37.853210 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6043d843-e92f-4964-97cf-5ab54b97a651 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:37.864281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6043d843-e92f-4964-97cf-5ab54b97a651 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:37.865977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6043d843-e92f-4964-97cf-5ab54b97a651 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:37.879468 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1171/2332] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:59:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:59:38.326299 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c919f1fe-ef7d-4c17-8b23-78a0aefa655b None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:38.328070 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c919f1fe-ef7d-4c17-8b23-78a0aefa655b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:38.355284 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c919f1fe-ef7d-4c17-8b23-78a0aefa655b None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:38.373323 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfce6ea-6581-4728-87f1-e5d6389fcad3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:38.375459 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfce6ea-6581-4728-87f1-e5d6389fcad3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:38.376556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfce6ea-6581-4728-87f1-e5d6389fcad3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:38.377618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfce6ea-6581-4728-87f1-e5d6389fcad3 admin admin] PATH_INFO: `/v3/projects/071b63e294fe42b5a26c6eaeecbd204d` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:38.381414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfce6ea-6581-4728-87f1-e5d6389fcad3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=071b63e294fe42b5a26c6eaeecbd204d)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:38.388706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfce6ea-6581-4728-87f1-e5d6389fcad3 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:38.527275 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9bfce6ea-6581-4728-87f1-e5d6389fcad3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '071b63e294fe42b5a26c6eaeecbd204d', 'request_id': 'req-9bfce6ea-6581-4728-87f1-e5d6389fcad3'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:38.544866 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9bfce6ea-6581-4728-87f1-e5d6389fcad3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '071b63e294fe42b5a26c6eaeecbd204d', 'request_id': 'req-9bfce6ea-6581-4728-87f1-e5d6389fcad3'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:38.563116 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9bfce6ea-6581-4728-87f1-e5d6389fcad3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '071b63e294fe42b5a26c6eaeecbd204d', 'request_id': 'req-9bfce6ea-6581-4728-87f1-e5d6389fcad3'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:38.579696 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9bfce6ea-6581-4728-87f1-e5d6389fcad3 admin admin] The token cache is being invalidate because project 071b63e294fe42b5a26c6eaeecbd204d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:59:38.580498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9bfce6ea-6581-4728-87f1-e5d6389fcad3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:38.582043 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9bfce6ea-6581-4728-87f1-e5d6389fcad3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:38.584686 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1162/2333] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:59:38 2022] DELETE /identity/v3/projects/071b63e294fe42b5a26c6eaeecbd204d => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:38.588832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6043d843-e92f-4964-97cf-5ab54b97a651 service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:38.603187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6043d843-e92f-4964-97cf-5ab54b97a651 service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:38.691139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd6467b-b05f-43cc-8b60-cb71e638e3d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:38.691890 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd6467b-b05f-43cc-8b60-cb71e638e3d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:38.692807 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd6467b-b05f-43cc-8b60-cb71e638e3d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:38.693205 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd6467b-b05f-43cc-8b60-cb71e638e3d3 admin admin] PATH_INFO: `/v3/users/872749a884f140079a35d42a979d539d` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:38.694633 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd6467b-b05f-43cc-8b60-cb71e638e3d3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=872749a884f140079a35d42a979d539d)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:38.697119 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd6467b-b05f-43cc-8b60-cb71e638e3d3 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:38.783302 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8cd6467b-b05f-43cc-8b60-cb71e638e3d3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '872749a884f140079a35d42a979d539d'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:38.816212 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8cd6467b-b05f-43cc-8b60-cb71e638e3d3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '872749a884f140079a35d42a979d539d'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:38.836679 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8cd6467b-b05f-43cc-8b60-cb71e638e3d3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '872749a884f140079a35d42a979d539d'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:38.838427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8cd6467b-b05f-43cc-8b60-cb71e638e3d3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '872749a884f140079a35d42a979d539d'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:38.858787 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8cd6467b-b05f-43cc-8b60-cb71e638e3d3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '872749a884f140079a35d42a979d539d'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:38.880034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1172/2334] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:59:38 2022] DELETE /identity/v3/users/872749a884f140079a35d42a979d539d => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:39.256133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfce6ea-6581-4728-87f1-e5d6389fcad3 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:39.258328 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-9bfce6ea-6581-4728-87f1-e5d6389fcad3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:39.270877 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:39.287309 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfce6ea-6581-4728-87f1-e5d6389fcad3 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:39.361113 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91f721c-805e-4b11-9f10-04c2046752a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:39.362075 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91f721c-805e-4b11-9f10-04c2046752a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:39.362652 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91f721c-805e-4b11-9f10-04c2046752a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:39.363154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91f721c-805e-4b11-9f10-04c2046752a7 admin admin] PATH_INFO: `/v3/projects/2cde2b94cae2420f96f0c1121b815181` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:39.364431 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91f721c-805e-4b11-9f10-04c2046752a7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2cde2b94cae2420f96f0c1121b815181)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:39.366709 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91f721c-805e-4b11-9f10-04c2046752a7 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:39.459625 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b91f721c-805e-4b11-9f10-04c2046752a7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2cde2b94cae2420f96f0c1121b815181', 'request_id': 'req-b91f721c-805e-4b11-9f10-04c2046752a7'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:39.467243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b91f721c-805e-4b11-9f10-04c2046752a7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2cde2b94cae2420f96f0c1121b815181', 'request_id': 'req-b91f721c-805e-4b11-9f10-04c2046752a7'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:39.478289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b91f721c-805e-4b11-9f10-04c2046752a7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2cde2b94cae2420f96f0c1121b815181', 'request_id': 'req-b91f721c-805e-4b11-9f10-04c2046752a7'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:39.493757 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b91f721c-805e-4b11-9f10-04c2046752a7 admin admin] The token cache is being invalidate because project 2cde2b94cae2420f96f0c1121b815181 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:59:39.494111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b91f721c-805e-4b11-9f10-04c2046752a7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:39.495786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b91f721c-805e-4b11-9f10-04c2046752a7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:39.497504 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1163/2335] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:59:39 2022] DELETE /identity/v3/projects/2cde2b94cae2420f96f0c1121b815181 => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:39.501647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd6467b-b05f-43cc-8b60-cb71e638e3d3 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:39.504337 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-8cd6467b-b05f-43cc-8b60-cb71e638e3d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:39.529600 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd6467b-b05f-43cc-8b60-cb71e638e3d3 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:39.613231 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:39.630308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9137c960-5e46-4595-8f19-c06284e50ec1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:39.631022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9137c960-5e46-4595-8f19-c06284e50ec1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:39.631378 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9137c960-5e46-4595-8f19-c06284e50ec1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:39.632227 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9137c960-5e46-4595-8f19-c06284e50ec1 admin admin] PATH_INFO: `/v3/users/5ba9ef3d01b34035be12c373dbe76b5e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:39.633499 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9137c960-5e46-4595-8f19-c06284e50ec1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5ba9ef3d01b34035be12c373dbe76b5e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:39.635662 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9137c960-5e46-4595-8f19-c06284e50ec1 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:39.695594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-9137c960-5e46-4595-8f19-c06284e50ec1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5ba9ef3d01b34035be12c373dbe76b5e'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:39.704662 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-9137c960-5e46-4595-8f19-c06284e50ec1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5ba9ef3d01b34035be12c373dbe76b5e'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:39.719465 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-9137c960-5e46-4595-8f19-c06284e50ec1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5ba9ef3d01b34035be12c373dbe76b5e'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:39.720377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-9137c960-5e46-4595-8f19-c06284e50ec1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5ba9ef3d01b34035be12c373dbe76b5e'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:39.727877 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-9137c960-5e46-4595-8f19-c06284e50ec1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5ba9ef3d01b34035be12c373dbe76b5e'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:39.737535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1173/2336] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:59:39 2022] DELETE /identity/v3/users/5ba9ef3d01b34035be12c373dbe76b5e => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:40.295352 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91f721c-805e-4b11-9f10-04c2046752a7 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:40.297469 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-b91f721c-805e-4b11-9f10-04c2046752a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:40.337619 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91f721c-805e-4b11-9f10-04c2046752a7 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:40.485815 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8228b6-9665-41f8-9d02-089fa2a8f93c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:40.486609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8228b6-9665-41f8-9d02-089fa2a8f93c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:40.487116 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8228b6-9665-41f8-9d02-089fa2a8f93c admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:40.487599 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8228b6-9665-41f8-9d02-089fa2a8f93c admin admin] PATH_INFO: `/v3/projects/a2d7fb49579646f195f993081ce7ea06` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:40.488950 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8228b6-9665-41f8-9d02-089fa2a8f93c admin admin] RBAC: Authorizing `identity:delete_project(project_id=a2d7fb49579646f195f993081ce7ea06)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:40.492308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8228b6-9665-41f8-9d02-089fa2a8f93c admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:40.537164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:40.549854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-3f8228b6-9665-41f8-9d02-089fa2a8f93c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a2d7fb49579646f195f993081ce7ea06', 'request_id': 'req-3f8228b6-9665-41f8-9d02-089fa2a8f93c'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:40.556424 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-3f8228b6-9665-41f8-9d02-089fa2a8f93c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a2d7fb49579646f195f993081ce7ea06', 'request_id': 'req-3f8228b6-9665-41f8-9d02-089fa2a8f93c'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:40.564773 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-3f8228b6-9665-41f8-9d02-089fa2a8f93c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a2d7fb49579646f195f993081ce7ea06', 'request_id': 'req-3f8228b6-9665-41f8-9d02-089fa2a8f93c'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:40.577437 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-3f8228b6-9665-41f8-9d02-089fa2a8f93c admin admin] The token cache is being invalidate because project a2d7fb49579646f195f993081ce7ea06 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:59:40.577730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-3f8228b6-9665-41f8-9d02-089fa2a8f93c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:40.578393 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-3f8228b6-9665-41f8-9d02-089fa2a8f93c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:40.580006 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1164/2337] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:59:40 2022] DELETE /identity/v3/projects/a2d7fb49579646f195f993081ce7ea06 => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:40.585441 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4f16fd-eb96-4f3f-abc6-00260bfcedac None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:40.585731 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4f16fd-eb96-4f3f-abc6-00260bfcedac None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:40.586017 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4f16fd-eb96-4f3f-abc6-00260bfcedac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:40.607845 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-bc4f16fd-eb96-4f3f-abc6-00260bfcedac None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:40.608991 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-bc4f16fd-eb96-4f3f-abc6-00260bfcedac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:40.673280 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1174/2338] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:40.681686 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736db9b2-2eaf-4eda-b870-acc47301b436 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:40.681976 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736db9b2-2eaf-4eda-b870-acc47301b436 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:40.682253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736db9b2-2eaf-4eda-b870-acc47301b436 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:40.702921 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-736db9b2-2eaf-4eda-b870-acc47301b436 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:40.704164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-736db9b2-2eaf-4eda-b870-acc47301b436 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:40.713438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1165/2339] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:40.717747 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4f16fd-eb96-4f3f-abc6-00260bfcedac None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:40.728092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4f16fd-eb96-4f3f-abc6-00260bfcedac None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:40.745147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98a3673-2731-44ba-b32f-5c9c298d5f44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:40.745816 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98a3673-2731-44ba-b32f-5c9c298d5f44 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:40.746099 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98a3673-2731-44ba-b32f-5c9c298d5f44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:40.746368 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98a3673-2731-44ba-b32f-5c9c298d5f44 admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:40.747069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98a3673-2731-44ba-b32f-5c9c298d5f44 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:59:40.747420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98a3673-2731-44ba-b32f-5c9c298d5f44 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:40.749055 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98a3673-2731-44ba-b32f-5c9c298d5f44 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:40.766123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1175/2340] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:59:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:59:40.767534 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736db9b2-2eaf-4eda-b870-acc47301b436 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:40.780885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736db9b2-2eaf-4eda-b870-acc47301b436 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:40.785810 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9226a049-590d-4e9b-ae3c-23d3b7bc22f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:40.786521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9226a049-590d-4e9b-ae3c-23d3b7bc22f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:40.786837 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9226a049-590d-4e9b-ae3c-23d3b7bc22f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:40.787123 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9226a049-590d-4e9b-ae3c-23d3b7bc22f9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:40.788574 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9226a049-590d-4e9b-ae3c-23d3b7bc22f9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:40.791343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9226a049-590d-4e9b-ae3c-23d3b7bc22f9 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:40.827690 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1166/2341] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:59:40 2022] POST /identity/v3/projects => generated 383 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:59:40.832284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98a3673-2731-44ba-b32f-5c9c298d5f44 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:40.842760 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98a3673-2731-44ba-b32f-5c9c298d5f44 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:40.901491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:40.910317 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be25c9c-2e70-4c2a-9e0f-e80c2c0f9c11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:40.911705 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be25c9c-2e70-4c2a-9e0f-e80c2c0f9c11 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:40.912116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be25c9c-2e70-4c2a-9e0f-e80c2c0f9c11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:40.912465 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be25c9c-2e70-4c2a-9e0f-e80c2c0f9c11 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:40.913692 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be25c9c-2e70-4c2a-9e0f-e80c2c0f9c11 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:40.915727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be25c9c-2e70-4c2a-9e0f-e80c2c0f9c11 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:40.948863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1176/2342] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:59:40 2022] POST /identity/v3/users => generated 292 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:59:40.953375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9226a049-590d-4e9b-ae3c-23d3b7bc22f9 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:40.964605 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9226a049-590d-4e9b-ae3c-23d3b7bc22f9 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:40.969595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6805f488-8e28-49c4-a0ac-063154cddbb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:40.970230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6805f488-8e28-49c4-a0ac-063154cddbb7 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:40.970548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6805f488-8e28-49c4-a0ac-063154cddbb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:40.970928 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6805f488-8e28-49c4-a0ac-063154cddbb7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:40.971613 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6805f488-8e28-49c4-a0ac-063154cddbb7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:40.973409 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6805f488-8e28-49c4-a0ac-063154cddbb7 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:40.990452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1167/2343] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:40 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:59:40.995980 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be25c9c-2e70-4c2a-9e0f-e80c2c0f9c11 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:41.008248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be25c9c-2e70-4c2a-9e0f-e80c2c0f9c11 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:41.013187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8e2d6f-c3ae-4189-9b36-57ae9e42e381 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:41.013810 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8e2d6f-c3ae-4189-9b36-57ae9e42e381 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:41.014096 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8e2d6f-c3ae-4189-9b36-57ae9e42e381 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:41.014424 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8e2d6f-c3ae-4189-9b36-57ae9e42e381 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:41.015391 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8e2d6f-c3ae-4189-9b36-57ae9e42e381 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:41.017363 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8e2d6f-c3ae-4189-9b36-57ae9e42e381 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:41.030313 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1177/2344] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:40 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:59:41.034800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6805f488-8e28-49c4-a0ac-063154cddbb7 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:41.048381 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6805f488-8e28-49c4-a0ac-063154cddbb7 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:41.054974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c9a762-5634-43a4-b065-953bf471481d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:41.055690 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c9a762-5634-43a4-b065-953bf471481d admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:41.056595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c9a762-5634-43a4-b065-953bf471481d admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:41.056874 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c9a762-5634-43a4-b065-953bf471481d admin admin] PATH_INFO: `/v3/projects/c9f737f6350f4d1f993708b1f90a862e/users/20a50c6db67f4d818a6fbeaafce299d9/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:41.073853 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8c9a762-5634-43a4-b065-953bf471481d admin admin] RBAC: Authorizing `identity:create_grant(project_id=c9f737f6350f4d1f993708b1f90a862e, user_id=20a50c6db67f4d818a6fbeaafce299d9, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:41.076798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8c9a762-5634-43a4-b065-953bf471481d admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:41.092258 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1168/2345] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:59:41 2022] PUT /identity/v3/projects/c9f737f6350f4d1f993708b1f90a862e/users/20a50c6db67f4d818a6fbeaafce299d9/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:41.099073 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202d23d4-21e6-4d4a-b129-429f0cf9fcfd None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:41.099361 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202d23d4-21e6-4d4a-b129-429f0cf9fcfd None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:41.099638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202d23d4-21e6-4d4a-b129-429f0cf9fcfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:41.119777 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-202d23d4-21e6-4d4a-b129-429f0cf9fcfd None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:41.121246 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-202d23d4-21e6-4d4a-b129-429f0cf9fcfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:41.187847 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1178/2346] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:41.200286 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c9a762-5634-43a4-b065-953bf471481d admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:41.203213 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c8c9a762-5634-43a4-b065-953bf471481d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:41.224428 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbf5eda-819a-429a-91d9-a3f667c91244 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:41.224927 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbf5eda-819a-429a-91d9-a3f667c91244 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:41.225224 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbf5eda-819a-429a-91d9-a3f667c91244 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:41.273722 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-0dbf5eda-819a-429a-91d9-a3f667c91244 None None] MFA Rules not processed for user `2f9751adf8d74ab59b6b0a5d00221405`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:41.275044 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-0dbf5eda-819a-429a-91d9-a3f667c91244 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:41.323815 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c9a762-5634-43a4-b065-953bf471481d admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:41.330338 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f6cb5d-d4d7-4f02-a224-692b6c4305e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:41.331240 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f6cb5d-d4d7-4f02-a224-692b6c4305e8 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:41.331766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f6cb5d-d4d7-4f02-a224-692b6c4305e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:41.332310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f6cb5d-d4d7-4f02-a224-692b6c4305e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:41.347759 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f6cb5d-d4d7-4f02-a224-692b6c4305e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:41.349378 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f6cb5d-d4d7-4f02-a224-692b6c4305e8 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:41.371191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1169/2347] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:59:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:59:41.408653 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1179/2348] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:41 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:41.719103 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb45824-7c2c-45e3-9187-97c981eaa298 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:41.719556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb45824-7c2c-45e3-9187-97c981eaa298 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:41.720014 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb45824-7c2c-45e3-9187-97c981eaa298 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:41.745227 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-dcb45824-7c2c-45e3-9187-97c981eaa298 None None] MFA Rules not processed for user `3e31dce69c344c9aa4a55f31b9c735df`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:41.746621 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-dcb45824-7c2c-45e3-9187-97c981eaa298 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:41.806215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1170/2349] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 18 00:59:41 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:41.860663 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbf5eda-819a-429a-91d9-a3f667c91244 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:41.862830 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-0dbf5eda-819a-429a-91d9-a3f667c91244 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:41.957280 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbf5eda-819a-429a-91d9-a3f667c91244 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:41.968312 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac501e2-fb6d-4f6c-b2a7-49b9068eddb0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6230ed7ce0eb43e9b2b8530a28eb6299', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:41.968958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac501e2-fb6d-4f6c-b2a7-49b9068eddb0 service swift] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:41.969239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac501e2-fb6d-4f6c-b2a7-49b9068eddb0 service swift] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:41.969552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac501e2-fb6d-4f6c-b2a7-49b9068eddb0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:41.979727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac501e2-fb6d-4f6c-b2a7-49b9068eddb0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:41.981577 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac501e2-fb6d-4f6c-b2a7-49b9068eddb0 service swift] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:41.990758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1180/2350] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 18 00:59:41 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 18 00:59:42.642068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb45824-7c2c-45e3-9187-97c981eaa298 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:42.643675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-dcb45824-7c2c-45e3-9187-97c981eaa298 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:42.687810 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb45824-7c2c-45e3-9187-97c981eaa298 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:42.703398 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eefeade-4c15-4086-b2b9-d5b220dcd125 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:42.704421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eefeade-4c15-4086-b2b9-d5b220dcd125 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:42.704715 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eefeade-4c15-4086-b2b9-d5b220dcd125 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:42.704983 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eefeade-4c15-4086-b2b9-d5b220dcd125 admin admin] PATH_INFO: `/v3/users/c9f9871f4c3e4a2c836e27edd2a52558` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:42.708518 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eefeade-4c15-4086-b2b9-d5b220dcd125 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c9f9871f4c3e4a2c836e27edd2a52558)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:42.714188 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eefeade-4c15-4086-b2b9-d5b220dcd125 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:42.717308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:DELETE=5,keystone:SELECT=92,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:42.814572 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9eefeade-4c15-4086-b2b9-d5b220dcd125 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c9f9871f4c3e4a2c836e27edd2a52558'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:42.828636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9eefeade-4c15-4086-b2b9-d5b220dcd125 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c9f9871f4c3e4a2c836e27edd2a52558'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:42.829538 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9eefeade-4c15-4086-b2b9-d5b220dcd125 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c9f9871f4c3e4a2c836e27edd2a52558'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:42.830589 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9eefeade-4c15-4086-b2b9-d5b220dcd125 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c9f9871f4c3e4a2c836e27edd2a52558'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:42.841869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9eefeade-4c15-4086-b2b9-d5b220dcd125 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c9f9871f4c3e4a2c836e27edd2a52558'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:42.896751 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1171/2351] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:59:42 2022] DELETE /identity/v3/users/c9f9871f4c3e4a2c836e27edd2a52558 => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:42.901598 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2824a61-e637-4b8f-9226-b05c32cea419 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:42.902151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2824a61-e637-4b8f-9226-b05c32cea419 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:42.902151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2824a61-e637-4b8f-9226-b05c32cea419 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:42.941321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-a2824a61-e637-4b8f-9226-b05c32cea419 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:42.943692 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a2824a61-e637-4b8f-9226-b05c32cea419 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:42.953723 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:43.083820 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1181/2352] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:43.097613 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eefeade-4c15-4086-b2b9-d5b220dcd125 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:43.099272 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-9eefeade-4c15-4086-b2b9-d5b220dcd125 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:43.264079 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eefeade-4c15-4086-b2b9-d5b220dcd125 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:43.275182 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74360355-5712-4951-8392-908d947dc39a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:43.283694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74360355-5712-4951-8392-908d947dc39a service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:43.285756 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74360355-5712-4951-8392-908d947dc39a service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:43.286177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74360355-5712-4951-8392-908d947dc39a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:43.310041 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74360355-5712-4951-8392-908d947dc39a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:43.311665 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74360355-5712-4951-8392-908d947dc39a service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:43.333517 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1172/2353] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:59:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 237 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:59:43.776959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2824a61-e637-4b8f-9226-b05c32cea419 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:43.778496 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a2824a61-e637-4b8f-9226-b05c32cea419 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:43.801250 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2824a61-e637-4b8f-9226-b05c32cea419 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:43.813445 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e547f3-b804-4377-bef1-0f342e6454ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:43.814177 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e547f3-b804-4377-bef1-0f342e6454ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:43.814528 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e547f3-b804-4377-bef1-0f342e6454ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:43.814813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e547f3-b804-4377-bef1-0f342e6454ab admin admin] PATH_INFO: `/v3/users/c33fc09990e0453e8c17c80ad660bc32` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:43.816610 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e547f3-b804-4377-bef1-0f342e6454ab admin admin] RBAC: Authorizing `identity:delete_user(user_id=c33fc09990e0453e8c17c80ad660bc32)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:43.823576 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e547f3-b804-4377-bef1-0f342e6454ab admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:43.893346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-b8e547f3-b804-4377-bef1-0f342e6454ab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c33fc09990e0453e8c17c80ad660bc32'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:43.907786 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-b8e547f3-b804-4377-bef1-0f342e6454ab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c33fc09990e0453e8c17c80ad660bc32'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:43.928836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-b8e547f3-b804-4377-bef1-0f342e6454ab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c33fc09990e0453e8c17c80ad660bc32'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:43.929655 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-b8e547f3-b804-4377-bef1-0f342e6454ab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c33fc09990e0453e8c17c80ad660bc32'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:43.972055 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-b8e547f3-b804-4377-bef1-0f342e6454ab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c33fc09990e0453e8c17c80ad660bc32'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:43.982314 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1182/2354] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 00:59:43 2022] DELETE /identity/v3/users/c33fc09990e0453e8c17c80ad660bc32 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:43.987356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092f8744-1308-4639-9c8e-46d9804fcd11 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:43.988881 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092f8744-1308-4639-9c8e-46d9804fcd11 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:43.989402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092f8744-1308-4639-9c8e-46d9804fcd11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:44.019635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-092f8744-1308-4639-9c8e-46d9804fcd11 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:44.023488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-092f8744-1308-4639-9c8e-46d9804fcd11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:44.186917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1173/2355] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:59:44.208371 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e547f3-b804-4377-bef1-0f342e6454ab admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:44.210341 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-b8e547f3-b804-4377-bef1-0f342e6454ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:44.323601 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e547f3-b804-4377-bef1-0f342e6454ab admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:44.329187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224af38b-a4e1-42a7-95cb-270215b0e09c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:44.330034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224af38b-a4e1-42a7-95cb-270215b0e09c service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:44.330799 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224af38b-a4e1-42a7-95cb-270215b0e09c service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:44.330799 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224af38b-a4e1-42a7-95cb-270215b0e09c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:44.365044 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224af38b-a4e1-42a7-95cb-270215b0e09c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:44.370040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224af38b-a4e1-42a7-95cb-270215b0e09c service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:44.395227 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092f8744-1308-4639-9c8e-46d9804fcd11 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:44.403077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-092f8744-1308-4639-9c8e-46d9804fcd11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:44.430953 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1183/2356] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:59:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:59:44.449691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092f8744-1308-4639-9c8e-46d9804fcd11 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:44.456660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0461c2d2-10f0-489b-b1ac-e3b67b148193 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:44.457694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0461c2d2-10f0-489b-b1ac-e3b67b148193 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:44.460751 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0461c2d2-10f0-489b-b1ac-e3b67b148193 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:44.461903 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0461c2d2-10f0-489b-b1ac-e3b67b148193 admin admin] PATH_INFO: `/v3/projects/faf829449e074324901c08d9e1215533` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:44.468351 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0461c2d2-10f0-489b-b1ac-e3b67b148193 admin admin] RBAC: Authorizing `identity:delete_project(project_id=faf829449e074324901c08d9e1215533)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:44.472352 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0461c2d2-10f0-489b-b1ac-e3b67b148193 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:44.517440 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:44.612813 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0461c2d2-10f0-489b-b1ac-e3b67b148193 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'faf829449e074324901c08d9e1215533', 'request_id': 'req-0461c2d2-10f0-489b-b1ac-e3b67b148193'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:44.633387 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0461c2d2-10f0-489b-b1ac-e3b67b148193 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'faf829449e074324901c08d9e1215533', 'request_id': 'req-0461c2d2-10f0-489b-b1ac-e3b67b148193'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:44.676288 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0461c2d2-10f0-489b-b1ac-e3b67b148193 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'faf829449e074324901c08d9e1215533', 'request_id': 'req-0461c2d2-10f0-489b-b1ac-e3b67b148193'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:44.700603 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0461c2d2-10f0-489b-b1ac-e3b67b148193 admin admin] The token cache is being invalidate because project faf829449e074324901c08d9e1215533 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:59:44.700851 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0461c2d2-10f0-489b-b1ac-e3b67b148193 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:44.701851 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0461c2d2-10f0-489b-b1ac-e3b67b148193 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:44.703657 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1174/2357] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:59:44 2022] DELETE /identity/v3/projects/faf829449e074324901c08d9e1215533 => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:44.708990 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51b3352-424f-4161-b5f2-05139d026887 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:44.709280 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51b3352-424f-4161-b5f2-05139d026887 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:44.709557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51b3352-424f-4161-b5f2-05139d026887 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:44.765726 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-a51b3352-424f-4161-b5f2-05139d026887 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:44.778035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a51b3352-424f-4161-b5f2-05139d026887 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:44.876370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:45.034083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1e4fa7-a4fe-4d5d-baa1-f4e5cea399e5 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:45.034574 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1e4fa7-a4fe-4d5d-baa1-f4e5cea399e5 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:45.035090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1e4fa7-a4fe-4d5d-baa1-f4e5cea399e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:45.045220 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1184/2358] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 337 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:45.082188 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-1e1e4fa7-a4fe-4d5d-baa1-f4e5cea399e5 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:45.084818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-1e1e4fa7-a4fe-4d5d-baa1-f4e5cea399e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:45.104547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51b3352-424f-4161-b5f2-05139d026887 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:45.112356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1175/2359] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:45.117307 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51b3352-424f-4161-b5f2-05139d026887 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:45.122737 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0accca23-0239-4f90-b8d9-b1a0de68b697 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:45.123438 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0accca23-0239-4f90-b8d9-b1a0de68b697 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:45.123941 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0accca23-0239-4f90-b8d9-b1a0de68b697 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:45.124551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0accca23-0239-4f90-b8d9-b1a0de68b697 admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:45.125398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0accca23-0239-4f90-b8d9-b1a0de68b697 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:59:45.125892 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0accca23-0239-4f90-b8d9-b1a0de68b697 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:45.127756 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0accca23-0239-4f90-b8d9-b1a0de68b697 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:45.146835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1e4fa7-a4fe-4d5d-baa1-f4e5cea399e5 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:45.153068 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1185/2360] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:59:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:59:45.177257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1e4fa7-a4fe-4d5d-baa1-f4e5cea399e5 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:45.186365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1616fbe0-a5ec-4306-b251-1121418ae044 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:45.187188 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1616fbe0-a5ec-4306-b251-1121418ae044 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:45.188102 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1616fbe0-a5ec-4306-b251-1121418ae044 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:45.188472 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1616fbe0-a5ec-4306-b251-1121418ae044 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:45.194224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1616fbe0-a5ec-4306-b251-1121418ae044 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:45.205761 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1616fbe0-a5ec-4306-b251-1121418ae044 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:45.297008 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1176/2361] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:59:45 2022] POST /identity/v3/projects => generated 373 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:59:45.300872 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0accca23-0239-4f90-b8d9-b1a0de68b697 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:45.317315 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0accca23-0239-4f90-b8d9-b1a0de68b697 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:45.361670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1616fbe0-a5ec-4306-b251-1121418ae044 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:45.364182 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-1616fbe0-a5ec-4306-b251-1121418ae044 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:45.549264 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1616fbe0-a5ec-4306-b251-1121418ae044 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:45.570617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0e8fb0-2aab-4b6b-913b-b24073e0903d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:45.571452 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23270590-74db-48ba-a76b-8d8be40002c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:45.572103 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0e8fb0-2aab-4b6b-913b-b24073e0903d admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:45.573360 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0e8fb0-2aab-4b6b-913b-b24073e0903d admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:45.573830 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0e8fb0-2aab-4b6b-913b-b24073e0903d admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:45.574306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23270590-74db-48ba-a76b-8d8be40002c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:45.574815 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23270590-74db-48ba-a76b-8d8be40002c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:45.575922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0e8fb0-2aab-4b6b-913b-b24073e0903d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:45.576393 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23270590-74db-48ba-a76b-8d8be40002c7 admin admin] PATH_INFO: `/v3/projects/efccc3a3c8254b15bf40f003f37832af` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:45.580637 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23270590-74db-48ba-a76b-8d8be40002c7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=efccc3a3c8254b15bf40f003f37832af)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:45.582304 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0e8fb0-2aab-4b6b-913b-b24073e0903d admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:45.588794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23270590-74db-48ba-a76b-8d8be40002c7 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:45.649861 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1186/2362] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:59:45 2022] POST /identity/v3/users => generated 287 bytes in 349 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:59:45.655462 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0e8fb0-2aab-4b6b-913b-b24073e0903d admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:45.687320 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0e8fb0-2aab-4b6b-913b-b24073e0903d admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:45.699673 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d94a3e7-8380-430a-9e62-1b7a62ff6531 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:45.700458 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d94a3e7-8380-430a-9e62-1b7a62ff6531 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:45.701311 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d94a3e7-8380-430a-9e62-1b7a62ff6531 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:45.701812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d94a3e7-8380-430a-9e62-1b7a62ff6531 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:45.702574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d94a3e7-8380-430a-9e62-1b7a62ff6531 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:45.704546 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d94a3e7-8380-430a-9e62-1b7a62ff6531 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:45.719675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1187/2363] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:45 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:59:45.723620 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d94a3e7-8380-430a-9e62-1b7a62ff6531 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:45.736975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d94a3e7-8380-430a-9e62-1b7a62ff6531 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:45.752812 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-23270590-74db-48ba-a76b-8d8be40002c7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'efccc3a3c8254b15bf40f003f37832af', 'request_id': 'req-23270590-74db-48ba-a76b-8d8be40002c7'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:45.765540 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-23270590-74db-48ba-a76b-8d8be40002c7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'efccc3a3c8254b15bf40f003f37832af', 'request_id': 'req-23270590-74db-48ba-a76b-8d8be40002c7'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:45.786945 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-23270590-74db-48ba-a76b-8d8be40002c7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'efccc3a3c8254b15bf40f003f37832af', 'request_id': 'req-23270590-74db-48ba-a76b-8d8be40002c7'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:45.815184 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-23270590-74db-48ba-a76b-8d8be40002c7 admin admin] The token cache is being invalidate because project efccc3a3c8254b15bf40f003f37832af was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 00:59:45.831526 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-23270590-74db-48ba-a76b-8d8be40002c7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:45.832686 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-23270590-74db-48ba-a76b-8d8be40002c7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 00:59:45.841326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1177/2364] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 00:59:45 2022] DELETE /identity/v3/projects/efccc3a3c8254b15bf40f003f37832af => generated 0 bytes in 474 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:45.843374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f860c9cc-6dcc-4626-980d-82e9d776543a None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:45.843888 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f860c9cc-6dcc-4626-980d-82e9d776543a None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:45.849122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f860c9cc-6dcc-4626-980d-82e9d776543a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:45.864335 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a8916e-343a-4786-b6d5-4b08db023cb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:45.865081 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a8916e-343a-4786-b6d5-4b08db023cb6 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:45.865450 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a8916e-343a-4786-b6d5-4b08db023cb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:45.865856 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a8916e-343a-4786-b6d5-4b08db023cb6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:45.867184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a8916e-343a-4786-b6d5-4b08db023cb6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:45.868636 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a8916e-343a-4786-b6d5-4b08db023cb6 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:45.883834 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:45.893614 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1188/2365] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:45 2022] GET /identity/v3/roles => generated 1420 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 00:59:45.898932 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a8916e-343a-4786-b6d5-4b08db023cb6 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:45.903436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-44a8916e-343a-4786-b6d5-4b08db023cb6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:45.908740 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-f860c9cc-6dcc-4626-980d-82e9d776543a None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:45.927516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-f860c9cc-6dcc-4626-980d-82e9d776543a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:45.934853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a8916e-343a-4786-b6d5-4b08db023cb6 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:45.947825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef04233-96fb-434a-82d1-7fe448b911b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:45.951136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef04233-96fb-434a-82d1-7fe448b911b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:45.951309 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef04233-96fb-434a-82d1-7fe448b911b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:45.951640 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef04233-96fb-434a-82d1-7fe448b911b8 admin admin] PATH_INFO: `/v3/projects/06165f0a9530412caf1e3454cfd0ca06/users/ba90a2f2147d4512a09c2f4685c9a510/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:45.985689 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1178/2366] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:45.994544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef04233-96fb-434a-82d1-7fe448b911b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=06165f0a9530412caf1e3454cfd0ca06, user_id=ba90a2f2147d4512a09c2f4685c9a510, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:45.996086 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3126bab4-78a3-4338-b509-574521e27bdd None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:46.000201 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef04233-96fb-434a-82d1-7fe448b911b8 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:46.003540 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3126bab4-78a3-4338-b509-574521e27bdd None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:46.004019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3126bab4-78a3-4338-b509-574521e27bdd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:46.032714 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1189/2367] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:59:45 2022] PUT /identity/v3/projects/06165f0a9530412caf1e3454cfd0ca06/users/ba90a2f2147d4512a09c2f4685c9a510/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:46.044340 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d02bd24-3283-4d24-a550-5ac60752c314 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:46.044749 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d02bd24-3283-4d24-a550-5ac60752c314 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:46.045045 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d02bd24-3283-4d24-a550-5ac60752c314 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:46.061521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:46.083088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-3126bab4-78a3-4338-b509-574521e27bdd None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:46.090077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-3126bab4-78a3-4338-b509-574521e27bdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:46.111517 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-7d02bd24-3283-4d24-a550-5ac60752c314 None None] MFA Rules not processed for user `ba90a2f2147d4512a09c2f4685c9a510`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:46.113569 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-7d02bd24-3283-4d24-a550-5ac60752c314 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:46.226950 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1179/2368] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:46.232240 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3126bab4-78a3-4338-b509-574521e27bdd None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:46.244536 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3126bab4-78a3-4338-b509-574521e27bdd None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:46.249746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34655a45-09f0-42f8-9404-a2356d37db02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:46.250960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34655a45-09f0-42f8-9404-a2356d37db02 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:46.251356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34655a45-09f0-42f8-9404-a2356d37db02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:46.251843 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34655a45-09f0-42f8-9404-a2356d37db02 admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:46.252840 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34655a45-09f0-42f8-9404-a2356d37db02 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 00:59:46.257932 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34655a45-09f0-42f8-9404-a2356d37db02 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:46.259848 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34655a45-09f0-42f8-9404-a2356d37db02 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:46.283404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1190/2369] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:46 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:46.308563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d02bd24-3283-4d24-a550-5ac60752c314 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:46.310105 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-7d02bd24-3283-4d24-a550-5ac60752c314 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:46.321788 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1180/2370] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 00:59:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 00:59:46.323486 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34655a45-09f0-42f8-9404-a2356d37db02 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:46.363274 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34655a45-09f0-42f8-9404-a2356d37db02 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:46.378570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b287648-a1a9-4d32-aff7-eeb4918609f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:46.385929 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b287648-a1a9-4d32-aff7-eeb4918609f0 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:46.386296 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b287648-a1a9-4d32-aff7-eeb4918609f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:46.386599 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b287648-a1a9-4d32-aff7-eeb4918609f0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:46.387333 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b287648-a1a9-4d32-aff7-eeb4918609f0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:46.394132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b287648-a1a9-4d32-aff7-eeb4918609f0 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:46.453886 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1181/2371] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:59:46 2022] POST /identity/v3/projects => generated 379 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:59:46.458153 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b287648-a1a9-4d32-aff7-eeb4918609f0 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:46.490350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d02bd24-3283-4d24-a550-5ac60752c314 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:46.642568 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caff3c84-93cf-418d-914b-18596ac96582 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:46.643211 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caff3c84-93cf-418d-914b-18596ac96582 service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:46.643490 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caff3c84-93cf-418d-914b-18596ac96582 service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:46.643763 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caff3c84-93cf-418d-914b-18596ac96582 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:46.656310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b287648-a1a9-4d32-aff7-eeb4918609f0 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:46.673800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b18a22-a70b-4a7d-9327-72362385b656 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:46.674448 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b18a22-a70b-4a7d-9327-72362385b656 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:46.674805 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b18a22-a70b-4a7d-9327-72362385b656 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:46.675087 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b18a22-a70b-4a7d-9327-72362385b656 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:46.675853 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b18a22-a70b-4a7d-9327-72362385b656 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:46.677815 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b18a22-a70b-4a7d-9327-72362385b656 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:46.753449 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1182/2372] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:59:46 2022] POST /identity/v3/users => generated 290 bytes in 295 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:59:46.754610 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b18a22-a70b-4a7d-9327-72362385b656 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:46.783136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b18a22-a70b-4a7d-9327-72362385b656 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:46.798106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5b75c1-ced2-4063-81e3-aa43536c7b23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:46.799068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5b75c1-ced2-4063-81e3-aa43536c7b23 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:46.799515 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5b75c1-ced2-4063-81e3-aa43536c7b23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:46.800399 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5b75c1-ced2-4063-81e3-aa43536c7b23 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:46.801609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5b75c1-ced2-4063-81e3-aa43536c7b23 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:46.804868 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5b75c1-ced2-4063-81e3-aa43536c7b23 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:46.846212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1183/2373] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:46 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:59:46.848398 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5b75c1-ced2-4063-81e3-aa43536c7b23 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:46.863967 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5b75c1-ced2-4063-81e3-aa43536c7b23 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:46.900851 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0000ea4e-6847-43fa-b050-8e354214db30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:46.901501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0000ea4e-6847-43fa-b050-8e354214db30 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:46.901783 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0000ea4e-6847-43fa-b050-8e354214db30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:46.902057 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0000ea4e-6847-43fa-b050-8e354214db30 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:46.902790 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0000ea4e-6847-43fa-b050-8e354214db30 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:46.904427 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0000ea4e-6847-43fa-b050-8e354214db30 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:46.909826 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caff3c84-93cf-418d-914b-18596ac96582 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:46.913928 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caff3c84-93cf-418d-914b-18596ac96582 service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:46.940689 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1184/2374] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:46 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:59:46.941991 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0000ea4e-6847-43fa-b050-8e354214db30 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:46.946884 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1191/2375] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:59:46 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 639 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 00:59:46.960803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0000ea4e-6847-43fa-b050-8e354214db30 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:46.969085 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148ee865-5dca-4450-b5d0-89d69c842d0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:46.973312 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148ee865-5dca-4450-b5d0-89d69c842d0b admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:46.974409 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148ee865-5dca-4450-b5d0-89d69c842d0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:46.974719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148ee865-5dca-4450-b5d0-89d69c842d0b admin admin] PATH_INFO: `/v3/projects/6958209222b44905bcb4baa1b8c26c58/users/823fa0c3eeeb489591c639b81afc02bc/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:47.013375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148ee865-5dca-4450-b5d0-89d69c842d0b admin admin] RBAC: Authorizing `identity:create_grant(project_id=6958209222b44905bcb4baa1b8c26c58, user_id=823fa0c3eeeb489591c639b81afc02bc, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:47.017015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148ee865-5dca-4450-b5d0-89d69c842d0b admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:47.073632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1185/2376] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:59:46 2022] PUT /identity/v3/projects/6958209222b44905bcb4baa1b8c26c58/users/823fa0c3eeeb489591c639b81afc02bc/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:47.079072 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38696214-ca9f-41b8-9978-2b1dd68a9c21 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:47.080882 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38696214-ca9f-41b8-9978-2b1dd68a9c21 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:47.082733 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38696214-ca9f-41b8-9978-2b1dd68a9c21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:47.101046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:47.138807 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-38696214-ca9f-41b8-9978-2b1dd68a9c21 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:47.140048 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-38696214-ca9f-41b8-9978-2b1dd68a9c21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:47.350050 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1192/2377] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 270 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 00:59:47.426421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148ee865-5dca-4450-b5d0-89d69c842d0b admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:47.431760 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-148ee865-5dca-4450-b5d0-89d69c842d0b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:47.459276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:47.608048 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148ee865-5dca-4450-b5d0-89d69c842d0b admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:47.632245 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed168449-17a9-4932-a178-7d1298706f7a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:47.633359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed168449-17a9-4932-a178-7d1298706f7a service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:47.634728 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed168449-17a9-4932-a178-7d1298706f7a service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:47.640393 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed168449-17a9-4932-a178-7d1298706f7a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:47.704578 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed168449-17a9-4932-a178-7d1298706f7a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:47.715049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed168449-17a9-4932-a178-7d1298706f7a service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:47.769505 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1186/2378] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 00:59:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 00:59:52.385881 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676bed5e-d77f-4596-ac03-af733799c707 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:52.386849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676bed5e-d77f-4596-ac03-af733799c707 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:52.387277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676bed5e-d77f-4596-ac03-af733799c707 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:52.437322 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-676bed5e-d77f-4596-ac03-af733799c707 None None] MFA Rules not processed for user `20a50c6db67f4d818a6fbeaafce299d9`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 00:59:52.438954 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-676bed5e-d77f-4596-ac03-af733799c707 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:52.590659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1193/2379] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 00:59:52 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 00:59:52.595700 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed168449-17a9-4932-a178-7d1298706f7a service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:52.598722 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ed168449-17a9-4932-a178-7d1298706f7a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 00:59:52.629625 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed168449-17a9-4932-a178-7d1298706f7a service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:52.637011 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b85363-782d-43fb-bf9b-2a974c2a880f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:52.638003 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b85363-782d-43fb-bf9b-2a974c2a880f admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:52.638559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b85363-782d-43fb-bf9b-2a974c2a880f admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:52.639156 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b85363-782d-43fb-bf9b-2a974c2a880f admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:52.642608 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b85363-782d-43fb-bf9b-2a974c2a880f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:52.644680 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b85363-782d-43fb-bf9b-2a974c2a880f admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:52.802740 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1187/2380] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 00:59:52 2022] POST /identity/v3/projects => generated 383 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:59:52.807564 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676bed5e-d77f-4596-ac03-af733799c707 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:52.823257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676bed5e-d77f-4596-ac03-af733799c707 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:53.029066 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551e873d-2d68-49ae-87e6-ef53447ae90f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:53.029964 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551e873d-2d68-49ae-87e6-ef53447ae90f admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:53.030394 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551e873d-2d68-49ae-87e6-ef53447ae90f admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:53.030881 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551e873d-2d68-49ae-87e6-ef53447ae90f admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:53.031862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-551e873d-2d68-49ae-87e6-ef53447ae90f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:53.034112 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-551e873d-2d68-49ae-87e6-ef53447ae90f admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:53.078660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1194/2381] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 00:59:52 2022] POST /identity/v3/users => generated 298 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 00:59:53.080132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b85363-782d-43fb-bf9b-2a974c2a880f admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:53.096498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b85363-782d-43fb-bf9b-2a974c2a880f admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:53.105850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260a0149-455d-4f1f-8987-7495847678d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:53.106790 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260a0149-455d-4f1f-8987-7495847678d2 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:53.107173 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260a0149-455d-4f1f-8987-7495847678d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:53.107580 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260a0149-455d-4f1f-8987-7495847678d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:53.120401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260a0149-455d-4f1f-8987-7495847678d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:53.122346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260a0149-455d-4f1f-8987-7495847678d2 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:53.138322 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1188/2382] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:53 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:59:53.142175 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551e873d-2d68-49ae-87e6-ef53447ae90f admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:53.161256 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551e873d-2d68-49ae-87e6-ef53447ae90f admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:53.216875 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062586bb-7a34-48e1-b981-d3235b90cd52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:53.217620 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062586bb-7a34-48e1-b981-d3235b90cd52 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:53.217899 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062586bb-7a34-48e1-b981-d3235b90cd52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:53.218171 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062586bb-7a34-48e1-b981-d3235b90cd52 admin admin] PATH_INFO: `/v3/projects/60700a7ed187472bb91e8e138940f65d/users/55df034b429644f2bc2ccd0f9f613f37/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:53.276085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062586bb-7a34-48e1-b981-d3235b90cd52 admin admin] RBAC: Authorizing `identity:create_grant(project_id=60700a7ed187472bb91e8e138940f65d, user_id=55df034b429644f2bc2ccd0f9f613f37, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:53.282060 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062586bb-7a34-48e1-b981-d3235b90cd52 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:53.329437 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 00:59:53.340546 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1195/2383] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:59:53 2022] PUT /identity/v3/projects/60700a7ed187472bb91e8e138940f65d/users/55df034b429644f2bc2ccd0f9f613f37/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 00:59:53.344460 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260a0149-455d-4f1f-8987-7495847678d2 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:53.357852 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260a0149-455d-4f1f-8987-7495847678d2 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:53.596703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c738dbc-1f3f-4248-9393-94ebc0df9a5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:53.597401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c738dbc-1f3f-4248-9393-94ebc0df9a5e admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:53.597636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c738dbc-1f3f-4248-9393-94ebc0df9a5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:53.597885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c738dbc-1f3f-4248-9393-94ebc0df9a5e admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:53.605588 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c738dbc-1f3f-4248-9393-94ebc0df9a5e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:53.607992 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c738dbc-1f3f-4248-9393-94ebc0df9a5e admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:53.632186 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1189/2384] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 00:59:53 2022] GET /identity/v3/roles => generated 1420 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 00:59:53.633075 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062586bb-7a34-48e1-b981-d3235b90cd52 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 00:59:53.654587 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062586bb-7a34-48e1-b981-d3235b90cd52 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 00:59:53.673827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e049947-e137-448f-b4ea-eadec4e553e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 00:59:53.677070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e049947-e137-448f-b4ea-eadec4e553e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 00:59:53.679423 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e049947-e137-448f-b4ea-eadec4e553e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 00:59:53.679887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e049947-e137-448f-b4ea-eadec4e553e4 admin admin] PATH_INFO: `/v3/projects/60700a7ed187472bb91e8e138940f65d/users/55df034b429644f2bc2ccd0f9f613f37/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 00:59:53.694914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e049947-e137-448f-b4ea-eadec4e553e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=60700a7ed187472bb91e8e138940f65d, user_id=55df034b429644f2bc2ccd0f9f613f37, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 00:59:53.700836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e049947-e137-448f-b4ea-eadec4e553e4 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 00:59:53.746778 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1196/2385] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 00:59:53 2022] PUT /identity/v3/projects/60700a7ed187472bb91e8e138940f65d/users/55df034b429644f2bc2ccd0f9f613f37/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:00:00.780200 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843fa09e-3215-4bc5-88d6-3250e4c33ad2 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:00.780818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843fa09e-3215-4bc5-88d6-3250e4c33ad2 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:00.780928 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843fa09e-3215-4bc5-88d6-3250e4c33ad2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:00.893990 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-843fa09e-3215-4bc5-88d6-3250e4c33ad2 None None] MFA Rules not processed for user `823fa0c3eeeb489591c639b81afc02bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:00:00.897917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-843fa09e-3215-4bc5-88d6-3250e4c33ad2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:01.092477 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1190/2386] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:00:00 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 316 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:00:01.096877 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e049947-e137-448f-b4ea-eadec4e553e4 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:00:01.214109 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e049947-e137-448f-b4ea-eadec4e553e4 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:00:01.224654 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b468e0-0132-43c3-8bdf-187950e65b4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:00:01.226575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b468e0-0132-43c3-8bdf-187950e65b4b admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:01.227765 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b468e0-0132-43c3-8bdf-187950e65b4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:01.229539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b468e0-0132-43c3-8bdf-187950e65b4b admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:01.231735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b468e0-0132-43c3-8bdf-187950e65b4b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:00:01.235717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b468e0-0132-43c3-8bdf-187950e65b4b admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:00:01.298430 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1197/2387] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:00:01 2022] POST /identity/v3/projects => generated 377 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 01:00:01.304211 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843fa09e-3215-4bc5-88d6-3250e4c33ad2 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:00:01.316969 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:00:01.327114 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843fa09e-3215-4bc5-88d6-3250e4c33ad2 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:00:01.428096 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a719f47-6e63-4f41-a71d-16b976debf51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:00:01.430438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a719f47-6e63-4f41-a71d-16b976debf51 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:01.430935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a719f47-6e63-4f41-a71d-16b976debf51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:01.431327 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a719f47-6e63-4f41-a71d-16b976debf51 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:01.432604 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a719f47-6e63-4f41-a71d-16b976debf51 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:00:01.435601 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a719f47-6e63-4f41-a71d-16b976debf51 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:00:01.479301 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1191/2388] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:00:01 2022] POST /identity/v3/users => generated 295 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:00:01.483388 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b468e0-0132-43c3-8bdf-187950e65b4b admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:00:01.501525 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b468e0-0132-43c3-8bdf-187950e65b4b admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:00:01.520152 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35f2e12-50b2-4846-a481-7c109854cfd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:00:01.528011 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35f2e12-50b2-4846-a481-7c109854cfd1 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:01.528537 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35f2e12-50b2-4846-a481-7c109854cfd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:01.528989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35f2e12-50b2-4846-a481-7c109854cfd1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:01.529852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35f2e12-50b2-4846-a481-7c109854cfd1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:00:01.531719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35f2e12-50b2-4846-a481-7c109854cfd1 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:00:01.567250 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1198/2389] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:00:01 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:00:01.568751 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a719f47-6e63-4f41-a71d-16b976debf51 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:00:01.581589 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a719f47-6e63-4f41-a71d-16b976debf51 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:00:01.607574 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb656406-11c4-484b-ae1f-5402f8a44a9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:00:01.609208 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb656406-11c4-484b-ae1f-5402f8a44a9b admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:01.611208 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb656406-11c4-484b-ae1f-5402f8a44a9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:01.612573 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb656406-11c4-484b-ae1f-5402f8a44a9b admin admin] PATH_INFO: `/v3/projects/bf2fa0f0609f48d2988f950b3878b5c0/users/d7ae4c525e6848b48d48d554bf36a644/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:01.644492 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb656406-11c4-484b-ae1f-5402f8a44a9b admin admin] RBAC: Authorizing `identity:create_grant(project_id=bf2fa0f0609f48d2988f950b3878b5c0, user_id=d7ae4c525e6848b48d48d554bf36a644, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:00:01.659789 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb656406-11c4-484b-ae1f-5402f8a44a9b admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:00:01.682800 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35f2e12-50b2-4846-a481-7c109854cfd1 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:00:01.683490 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1192/2390] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:00:01 2022] PUT /identity/v3/projects/bf2fa0f0609f48d2988f950b3878b5c0/users/d7ae4c525e6848b48d48d554bf36a644/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:00:01.695871 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35f2e12-50b2-4846-a481-7c109854cfd1 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:00:01.836323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cded4f95-1a6b-44d8-8ffc-0f9b2dbfed8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:00:01.838821 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cded4f95-1a6b-44d8-8ffc-0f9b2dbfed8d admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:01.840501 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cded4f95-1a6b-44d8-8ffc-0f9b2dbfed8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:01.842263 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cded4f95-1a6b-44d8-8ffc-0f9b2dbfed8d admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:01.843160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cded4f95-1a6b-44d8-8ffc-0f9b2dbfed8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:00:01.845881 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cded4f95-1a6b-44d8-8ffc-0f9b2dbfed8d admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:00:01.872385 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb656406-11c4-484b-ae1f-5402f8a44a9b admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:00:01.873395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1199/2391] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:00:01 2022] GET /identity/v3/roles => generated 1420 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:00:01.884946 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb656406-11c4-484b-ae1f-5402f8a44a9b admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:00:01.890066 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8cde83-1396-447c-84ac-b359153c53a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:00:01.891663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8cde83-1396-447c-84ac-b359153c53a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:01.892115 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8cde83-1396-447c-84ac-b359153c53a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:01.892551 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8cde83-1396-447c-84ac-b359153c53a7 admin admin] PATH_INFO: `/v3/projects/bf2fa0f0609f48d2988f950b3878b5c0/users/d7ae4c525e6848b48d48d554bf36a644/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:01.894724 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8cde83-1396-447c-84ac-b359153c53a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bf2fa0f0609f48d2988f950b3878b5c0, user_id=d7ae4c525e6848b48d48d554bf36a644, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:00:01.898581 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8cde83-1396-447c-84ac-b359153c53a7 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:00:01.918058 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1193/2392] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:00:01 2022] PUT /identity/v3/projects/bf2fa0f0609f48d2988f950b3878b5c0/users/d7ae4c525e6848b48d48d554bf36a644/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:00:03.535764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cded4f95-1a6b-44d8-8ffc-0f9b2dbfed8d admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:00:03.558947 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cded4f95-1a6b-44d8-8ffc-0f9b2dbfed8d admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:00:03.694747 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:00:03.705525 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f51020-a93a-4919-9917-38a51bb511d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:00:03.706659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f51020-a93a-4919-9917-38a51bb511d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:03.707176 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f51020-a93a-4919-9917-38a51bb511d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:03.707686 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f51020-a93a-4919-9917-38a51bb511d7 admin admin] PATH_INFO: `/v3/users/ba90a2f2147d4512a09c2f4685c9a510` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:03.710315 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f51020-a93a-4919-9917-38a51bb511d7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ba90a2f2147d4512a09c2f4685c9a510)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:00:03.715691 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f51020-a93a-4919-9917-38a51bb511d7 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:00:03.852381 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-93f51020-a93a-4919-9917-38a51bb511d7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ba90a2f2147d4512a09c2f4685c9a510'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:00:03.872117 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-93f51020-a93a-4919-9917-38a51bb511d7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ba90a2f2147d4512a09c2f4685c9a510'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:00:03.904651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-93f51020-a93a-4919-9917-38a51bb511d7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ba90a2f2147d4512a09c2f4685c9a510'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:00:03.905564 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-93f51020-a93a-4919-9917-38a51bb511d7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ba90a2f2147d4512a09c2f4685c9a510'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:00:03.929611 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-93f51020-a93a-4919-9917-38a51bb511d7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ba90a2f2147d4512a09c2f4685c9a510'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:00:03.950397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1200/2393] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:00:03 2022] DELETE /identity/v3/users/ba90a2f2147d4512a09c2f4685c9a510 => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:00:03.955576 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab06c497-3ad7-4a5f-b464-a793d3e26db3 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:03.956111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab06c497-3ad7-4a5f-b464-a793d3e26db3 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:03.956111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab06c497-3ad7-4a5f-b464-a793d3e26db3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:04.020189 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-ab06c497-3ad7-4a5f-b464-a793d3e26db3 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:00:04.021801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ab06c497-3ad7-4a5f-b464-a793d3e26db3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:04.255883 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1194/2394] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:00:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 302 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:00:04.302055 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f51020-a93a-4919-9917-38a51bb511d7 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:00:04.303964 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-93f51020-a93a-4919-9917-38a51bb511d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:04.513773 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f51020-a93a-4919-9917-38a51bb511d7 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:00:04.527069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31706ddf-1361-4b1e-b31a-c0757a66ef69 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:00:04.527751 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31706ddf-1361-4b1e-b31a-c0757a66ef69 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:04.528027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31706ddf-1361-4b1e-b31a-c0757a66ef69 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:04.528305 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31706ddf-1361-4b1e-b31a-c0757a66ef69 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:04.569101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31706ddf-1361-4b1e-b31a-c0757a66ef69 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:00:04.572371 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31706ddf-1361-4b1e-b31a-c0757a66ef69 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:00:04.623331 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1201/2395] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:00:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:00:05.896754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab06c497-3ad7-4a5f-b464-a793d3e26db3 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:00:05.901754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ab06c497-3ad7-4a5f-b464-a793d3e26db3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:05.929171 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab06c497-3ad7-4a5f-b464-a793d3e26db3 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:00:05.940330 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e17c28d-e0d0-410f-a52a-200b3755bc2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:00:05.942185 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e17c28d-e0d0-410f-a52a-200b3755bc2b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:05.942742 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e17c28d-e0d0-410f-a52a-200b3755bc2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:05.943534 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e17c28d-e0d0-410f-a52a-200b3755bc2b admin admin] PATH_INFO: `/v3/projects/06165f0a9530412caf1e3454cfd0ca06` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:05.948220 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e17c28d-e0d0-410f-a52a-200b3755bc2b admin admin] RBAC: Authorizing `identity:delete_project(project_id=06165f0a9530412caf1e3454cfd0ca06)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:00:05.950948 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e17c28d-e0d0-410f-a52a-200b3755bc2b admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:00:06.047008 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0e17c28d-e0d0-410f-a52a-200b3755bc2b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '06165f0a9530412caf1e3454cfd0ca06', 'request_id': 'req-0e17c28d-e0d0-410f-a52a-200b3755bc2b'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:00:06.051544 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=90,keystone:INSERT=5,keystone:DELETE=5 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:00:06.069806 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0e17c28d-e0d0-410f-a52a-200b3755bc2b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '06165f0a9530412caf1e3454cfd0ca06', 'request_id': 'req-0e17c28d-e0d0-410f-a52a-200b3755bc2b'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:00:06.085228 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0e17c28d-e0d0-410f-a52a-200b3755bc2b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '06165f0a9530412caf1e3454cfd0ca06', 'request_id': 'req-0e17c28d-e0d0-410f-a52a-200b3755bc2b'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:00:06.103379 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0e17c28d-e0d0-410f-a52a-200b3755bc2b admin admin] The token cache is being invalidate because project 06165f0a9530412caf1e3454cfd0ca06 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:00:06.103785 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0e17c28d-e0d0-410f-a52a-200b3755bc2b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:00:06.105236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0e17c28d-e0d0-410f-a52a-200b3755bc2b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:00:06.107666 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1195/2396] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:00:05 2022] DELETE /identity/v3/projects/06165f0a9530412caf1e3454cfd0ca06 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:00:06.592249 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06af6a76-aad9-4029-944a-63e8bc9e8a1b None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:06.592783 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06af6a76-aad9-4029-944a-63e8bc9e8a1b None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:06.592783 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06af6a76-aad9-4029-944a-63e8bc9e8a1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:06.645431 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-06af6a76-aad9-4029-944a-63e8bc9e8a1b None None] MFA Rules not processed for user `55df034b429644f2bc2ccd0f9f613f37`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:00:06.646935 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-06af6a76-aad9-4029-944a-63e8bc9e8a1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:06.805814 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1202/2397] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:00:06 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:00:06.813672 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e17c28d-e0d0-410f-a52a-200b3755bc2b admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:00:06.815206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-0e17c28d-e0d0-410f-a52a-200b3755bc2b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:06.956192 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e17c28d-e0d0-410f-a52a-200b3755bc2b admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:00:06.964194 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85342735-65a3-46b6-b776-679fe37860e3 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7fe9d21099e4c4891c28687bb673214', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:00:06.965167 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85342735-65a3-46b6-b776-679fe37860e3 service glance] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:06.965609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85342735-65a3-46b6-b776-679fe37860e3 service glance] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:06.966435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85342735-65a3-46b6-b776-679fe37860e3 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:06.969270 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-85342735-65a3-46b6-b776-679fe37860e3 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:07.146492 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85342735-65a3-46b6-b776-679fe37860e3 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:00:07.151178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85342735-65a3-46b6-b776-679fe37860e3 service glance] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:00:07.175594 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1196/2398] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:00:06 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:00:07.363605 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a116030-8ab5-4274-b3d1-d3b87e479d54 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:07.364031 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a116030-8ab5-4274-b3d1-d3b87e479d54 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:07.364411 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a116030-8ab5-4274-b3d1-d3b87e479d54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:07.382426 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:00:07.423669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-7a116030-8ab5-4274-b3d1-d3b87e479d54 None None] MFA Rules not processed for user `3e31dce69c344c9aa4a55f31b9c735df`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:00:07.429810 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-7a116030-8ab5-4274-b3d1-d3b87e479d54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:07.577468 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1203/2399] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 18 01:00:07 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:00:07.585549 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85342735-65a3-46b6-b776-679fe37860e3 service glance] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:00:07.588177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-85342735-65a3-46b6-b776-679fe37860e3 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:07.802091 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85342735-65a3-46b6-b776-679fe37860e3 service glance] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:00:07.866703 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb859d2-a765-465d-8d4d-953dd46dfa75 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6230ed7ce0eb43e9b2b8530a28eb6299', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:00:07.867339 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb859d2-a765-465d-8d4d-953dd46dfa75 service swift] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:07.867660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb859d2-a765-465d-8d4d-953dd46dfa75 service swift] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:07.867879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb859d2-a765-465d-8d4d-953dd46dfa75 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:07.923071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb859d2-a765-465d-8d4d-953dd46dfa75 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:00:07.924904 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb859d2-a765-465d-8d4d-953dd46dfa75 service swift] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:00:07.939262 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1197/2400] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 18 01:00:07 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 18 01:00:13.488670 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b873fd3e-d316-495a-b355-94b33232d64f None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:13.489157 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b873fd3e-d316-495a-b355-94b33232d64f None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:13.489238 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b873fd3e-d316-495a-b355-94b33232d64f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:13.549320 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-b873fd3e-d316-495a-b355-94b33232d64f None None] MFA Rules not processed for user `d7ae4c525e6848b48d48d554bf36a644`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:00:13.550928 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-b873fd3e-d316-495a-b355-94b33232d64f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:13.730367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1204/2401] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:00:13 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:00:13.735598 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb859d2-a765-465d-8d4d-953dd46dfa75 service swift] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:00:13.737653 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8fb859d2-a765-465d-8d4d-953dd46dfa75 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:13.793961 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:DELETE=2,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:00:13.905860 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb859d2-a765-465d-8d4d-953dd46dfa75 service swift] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:00:13.921238 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f7d2fc-511c-437c-8509-c120c66217b8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:00:13.921895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f7d2fc-511c-437c-8509-c120c66217b8 service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:13.922177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f7d2fc-511c-437c-8509-c120c66217b8 service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:13.922449 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f7d2fc-511c-437c-8509-c120c66217b8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:13.951215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-12f7d2fc-511c-437c-8509-c120c66217b8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:14.154069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f7d2fc-511c-437c-8509-c120c66217b8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:00:14.155747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f7d2fc-511c-437c-8509-c120c66217b8 service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:00:14.183258 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1198/2402] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:00:13 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 448 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:00:14.520443 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5a1517-cccc-4d26-bdac-510d950b325a None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:14.521057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5a1517-cccc-4d26-bdac-510d950b325a None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:14.521229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5a1517-cccc-4d26-bdac-510d950b325a None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:14.523324 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1205/2403] 10.222.0.42 () {54 vars in 961 bytes} [Sun Sep 18 01:00:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 01:00:14.532575 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb8a25e-36b8-4457-bff1-ff5476bf582d None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:14.533054 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb8a25e-36b8-4457-bff1-ff5476bf582d None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:14.533553 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb8a25e-36b8-4457-bff1-ff5476bf582d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:14.603289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-0eb8a25e-36b8-4457-bff1-ff5476bf582d None None] MFA Rules not processed for user `8f30116fb5ce43dab69efa8bafd9d049`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:00:14.606763 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-0eb8a25e-36b8-4457-bff1-ff5476bf582d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:14.764187 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1199/2404] 10.222.0.42 () {58 vars in 1058 bytes} [Sun Sep 18 01:00:14 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:00:14.771574 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5a1517-cccc-4d26-bdac-510d950b325a None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:00:14.791169 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5a1517-cccc-4d26-bdac-510d950b325a None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:00:14.798950 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eab19a5-e45c-47c8-98bb-1924c3fcdffb service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8f30116fb5ce43dab69efa8bafd9d049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:00:14.800019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eab19a5-e45c-47c8-98bb-1924c3fcdffb service placement] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:14.800461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eab19a5-e45c-47c8-98bb-1924c3fcdffb service placement] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:14.801083 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eab19a5-e45c-47c8-98bb-1924c3fcdffb service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:14.808728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-7eab19a5-e45c-47c8-98bb-1924c3fcdffb service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:14.824764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eab19a5-e45c-47c8-98bb-1924c3fcdffb service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:00:14.827256 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eab19a5-e45c-47c8-98bb-1924c3fcdffb service placement] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:00:14.845044 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1206/2405] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:00:14 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 74 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:00:17.316136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5b66de-c189-4b54-a039-5e8060188f68 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:17.317594 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5b66de-c189-4b54-a039-5e8060188f68 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:17.318083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5b66de-c189-4b54-a039-5e8060188f68 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:17.320632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1200/2406] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 18 01:00:17 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 01:00:17.339376 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5e3f15-1a28-4ee3-855d-e3da6c4061df None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:17.340472 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5e3f15-1a28-4ee3-855d-e3da6c4061df None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:17.341721 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5e3f15-1a28-4ee3-855d-e3da6c4061df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:17.403867 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-af5e3f15-1a28-4ee3-855d-e3da6c4061df None None] MFA Rules not processed for user `a59d73fd45a54651930d52300704fdb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:00:17.416609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-af5e3f15-1a28-4ee3-855d-e3da6c4061df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:17.481865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1207/2407] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 18 01:00:17 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:00:17.487831 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5b66de-c189-4b54-a039-5e8060188f68 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:00:17.489926 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-6e5b66de-c189-4b54-a039-5e8060188f68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:17.544202 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5b66de-c189-4b54-a039-5e8060188f68 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:00:17.558695 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1978dbe3-4183-4e4e-98cf-f807d97ed533 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7fe9d21099e4c4891c28687bb673214', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:00:17.561346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1978dbe3-4183-4e4e-98cf-f807d97ed533 service glance] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:17.561858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1978dbe3-4183-4e4e-98cf-f807d97ed533 service glance] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:17.570068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1978dbe3-4183-4e4e-98cf-f807d97ed533 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:17.597527 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1978dbe3-4183-4e4e-98cf-f807d97ed533 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:00:17.599195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1978dbe3-4183-4e4e-98cf-f807d97ed533 service glance] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:00:17.630674 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1201/2408] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:00:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 01:00:17.816530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59a386c-fe7f-4001-9282-f832d7fe1242 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:17.816824 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59a386c-fe7f-4001-9282-f832d7fe1242 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:17.817097 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59a386c-fe7f-4001-9282-f832d7fe1242 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:17.819834 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1208/2409] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 18 01:00:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 01:00:17.838906 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bdb9ee-805d-4839-9af7-d279bb900e6e None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:17.839187 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bdb9ee-805d-4839-9af7-d279bb900e6e None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:17.839651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bdb9ee-805d-4839-9af7-d279bb900e6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:17.879576 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-34bdb9ee-805d-4839-9af7-d279bb900e6e None None] MFA Rules not processed for user `9b0510812b60454586a781e6399e77a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:00:17.883209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-34bdb9ee-805d-4839-9af7-d279bb900e6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:17.988554 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:00:18.019690 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1202/2410] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 18 01:00:17 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:00:18.030265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59a386c-fe7f-4001-9282-f832d7fe1242 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:00:18.032134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-d59a386c-fe7f-4001-9282-f832d7fe1242 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:18.093061 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59a386c-fe7f-4001-9282-f832d7fe1242 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:00:18.106142 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45491d95-5857-407d-947f-0dc9226a1e72 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:00:18.108897 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45491d95-5857-407d-947f-0dc9226a1e72 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:18.109209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45491d95-5857-407d-947f-0dc9226a1e72 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:18.109486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45491d95-5857-407d-947f-0dc9226a1e72 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:18.145559 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45491d95-5857-407d-947f-0dc9226a1e72 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:00:18.147888 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45491d95-5857-407d-947f-0dc9226a1e72 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:00:18.167701 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1209/2411] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:00:18 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:00:18.381485 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4625648b-25b1-45a9-ae7a-225b4848d144 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:18.382056 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4625648b-25b1-45a9-ae7a-225b4848d144 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:18.382311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4625648b-25b1-45a9-ae7a-225b4848d144 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:18.383939 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1203/2412] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 18 01:00:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 01:00:18.392110 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f96abd-88f5-474f-9cc5-0947c39bcf35 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:18.392849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f96abd-88f5-474f-9cc5-0947c39bcf35 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:18.393388 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f96abd-88f5-474f-9cc5-0947c39bcf35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:18.427527 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-51f96abd-88f5-474f-9cc5-0947c39bcf35 None None] MFA Rules not processed for user `a59d73fd45a54651930d52300704fdb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:00:18.429328 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-51f96abd-88f5-474f-9cc5-0947c39bcf35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:18.442901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1210/2413] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 18 01:00:18 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:00:18.460956 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4625648b-25b1-45a9-ae7a-225b4848d144 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:00:18.478957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-4625648b-25b1-45a9-ae7a-225b4848d144 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:18.496149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4625648b-25b1-45a9-ae7a-225b4848d144 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:00:18.504206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d23cf32-bee7-4d58-bc5a-7a4c2c498c35 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:00:18.505122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d23cf32-bee7-4d58-bc5a-7a4c2c498c35 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:18.505335 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d23cf32-bee7-4d58-bc5a-7a4c2c498c35 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:18.505605 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d23cf32-bee7-4d58-bc5a-7a4c2c498c35 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:18.540403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d23cf32-bee7-4d58-bc5a-7a4c2c498c35 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:00:18.542634 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d23cf32-bee7-4d58-bc5a-7a4c2c498c35 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:00:18.562219 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1204/2414] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:00:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:00:24.775579 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83bbc1c3-011a-4ec1-a428-7bc2af6cb73e None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:24.776169 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83bbc1c3-011a-4ec1-a428-7bc2af6cb73e None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:24.776331 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83bbc1c3-011a-4ec1-a428-7bc2af6cb73e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:24.796008 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-83bbc1c3-011a-4ec1-a428-7bc2af6cb73e None None] MFA Rules not processed for user `20a50c6db67f4d818a6fbeaafce299d9`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:00:24.797211 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-83bbc1c3-011a-4ec1-a428-7bc2af6cb73e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:24.805650 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1211/2415] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:00:24 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:00:24.911406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce59bfa8-4903-4402-b71b-14898c6acf5b None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:24.911933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce59bfa8-4903-4402-b71b-14898c6acf5b None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:24.911933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce59bfa8-4903-4402-b71b-14898c6acf5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:24.932355 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-ce59bfa8-4903-4402-b71b-14898c6acf5b None None] MFA Rules not processed for user `3e31dce69c344c9aa4a55f31b9c735df`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:00:24.933597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ce59bfa8-4903-4402-b71b-14898c6acf5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:24.941440 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1205/2416] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 18 01:00:24 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:00:24.954014 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83bbc1c3-011a-4ec1-a428-7bc2af6cb73e None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:00:24.966129 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83bbc1c3-011a-4ec1-a428-7bc2af6cb73e None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:00:24.971202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354f715d-2c7c-4b62-a7a0-af4f34a51fdb service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6230ed7ce0eb43e9b2b8530a28eb6299', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:00:24.971804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354f715d-2c7c-4b62-a7a0-af4f34a51fdb service swift] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:24.972119 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354f715d-2c7c-4b62-a7a0-af4f34a51fdb service swift] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:24.972390 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354f715d-2c7c-4b62-a7a0-af4f34a51fdb service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:24.985981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354f715d-2c7c-4b62-a7a0-af4f34a51fdb service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:00:24.987658 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354f715d-2c7c-4b62-a7a0-af4f34a51fdb service swift] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:00:24.996955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1212/2417] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 18 01:00:24 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 18 01:00:33.376919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fd2945-c659-4e6c-ab47-b3afb5361084 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:33.377549 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fd2945-c659-4e6c-ab47-b3afb5361084 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:33.377549 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fd2945-c659-4e6c-ab47-b3afb5361084 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:33.396607 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-c7fd2945-c659-4e6c-ab47-b3afb5361084 None None] MFA Rules not processed for user `823fa0c3eeeb489591c639b81afc02bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:00:33.398545 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c7fd2945-c659-4e6c-ab47-b3afb5361084 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:33.409112 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1206/2418] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:00:33 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:00:34.993466 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=84 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:00:36.280383 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8065e81c-34f9-439e-879b-b831fc9bdf4e None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:36.281048 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8065e81c-34f9-439e-879b-b831fc9bdf4e None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:36.281048 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8065e81c-34f9-439e-879b-b831fc9bdf4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:36.300509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-8065e81c-34f9-439e-879b-b831fc9bdf4e None None] MFA Rules not processed for user `3e31dce69c344c9aa4a55f31b9c735df`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:00:36.302016 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-8065e81c-34f9-439e-879b-b831fc9bdf4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:36.310804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1213/2419] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 18 01:00:36 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:00:36.325694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fd2945-c659-4e6c-ab47-b3afb5361084 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:00:36.342121 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fd2945-c659-4e6c-ab47-b3afb5361084 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:00:36.348166 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed65b0d-25bc-4010-8e19-abea9db5b237 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6230ed7ce0eb43e9b2b8530a28eb6299', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:00:36.348817 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed65b0d-25bc-4010-8e19-abea9db5b237 service swift] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:36.349088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed65b0d-25bc-4010-8e19-abea9db5b237 service swift] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:36.349520 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed65b0d-25bc-4010-8e19-abea9db5b237 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:36.359198 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed65b0d-25bc-4010-8e19-abea9db5b237 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:00:36.360825 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed65b0d-25bc-4010-8e19-abea9db5b237 service swift] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:00:36.377444 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1207/2420] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 18 01:00:36 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 18 01:00:46.295089 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:00:46.366795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=21 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:00:56.302562 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c7ed5b-c09d-4fd5-864a-c76e939af9f8 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:56.303474 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c7ed5b-c09d-4fd5-864a-c76e939af9f8 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:56.303750 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c7ed5b-c09d-4fd5-864a-c76e939af9f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:56.324143 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-47c7ed5b-c09d-4fd5-864a-c76e939af9f8 None None] MFA Rules not processed for user `3e31dce69c344c9aa4a55f31b9c735df`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:00:56.325308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-47c7ed5b-c09d-4fd5-864a-c76e939af9f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:00:56.332992 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1214/2421] 10.222.0.42 () {58 vars in 967 bytes} [Sun Sep 18 01:00:56 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:00:56.356211 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed65b0d-25bc-4010-8e19-abea9db5b237 service swift] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:00:56.368043 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed65b0d-25bc-4010-8e19-abea9db5b237 service swift] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:00:56.373259 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2796ebc-598c-48e1-b4d9-4d1be9f736de service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6230ed7ce0eb43e9b2b8530a28eb6299', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:00:56.373854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2796ebc-598c-48e1-b4d9-4d1be9f736de service swift] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:00:56.374173 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2796ebc-598c-48e1-b4d9-4d1be9f736de service swift] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:00:56.374446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2796ebc-598c-48e1-b4d9-4d1be9f736de service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:00:56.382873 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2796ebc-598c-48e1-b4d9-4d1be9f736de service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:00:56.384500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2796ebc-598c-48e1-b4d9-4d1be9f736de service swift] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:00:56.392901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1208/2422] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 18 01:00:56 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 18 01:01:06.318222 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:01:06.389414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=6 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:01:09.869817 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c7ed5b-c09d-4fd5-864a-c76e939af9f8 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:01:09.879068 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c7ed5b-c09d-4fd5-864a-c76e939af9f8 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:01:09.882600 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c55f1e-de51-4489-a49b-b648dc0acd5b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:01:09.883197 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c55f1e-de51-4489-a49b-b648dc0acd5b service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:01:09.883473 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c55f1e-de51-4489-a49b-b648dc0acd5b service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:01:09.883744 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c55f1e-de51-4489-a49b-b648dc0acd5b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:01:09.892569 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c55f1e-de51-4489-a49b-b648dc0acd5b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:01:09.894004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c55f1e-de51-4489-a49b-b648dc0acd5b service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:01:09.903419 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1215/2423] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:01:09 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:01:11.836007 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2796ebc-598c-48e1-b4d9-4d1be9f736de service swift] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:01:11.852808 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2796ebc-598c-48e1-b4d9-4d1be9f736de service swift] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:01:11.866759 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deff3ccc-521a-44b8-822f-f34d36132636 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:01:11.867367 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deff3ccc-521a-44b8-822f-f34d36132636 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:01:11.867645 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deff3ccc-521a-44b8-822f-f34d36132636 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:01:11.867928 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deff3ccc-521a-44b8-822f-f34d36132636 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:01:11.878178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deff3ccc-521a-44b8-822f-f34d36132636 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:01:11.880614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deff3ccc-521a-44b8-822f-f34d36132636 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:01:11.910879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1209/2424] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:01:11 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 76 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:01:18.738926 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b75d11-11cd-49ac-8028-20b369b07cf1 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:01:18.739454 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b75d11-11cd-49ac-8028-20b369b07cf1 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:01:18.739565 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b75d11-11cd-49ac-8028-20b369b07cf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:01:18.759204 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-d2b75d11-11cd-49ac-8028-20b369b07cf1 None None] MFA Rules not processed for user `2f9751adf8d74ab59b6b0a5d00221405`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:01:18.760385 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-d2b75d11-11cd-49ac-8028-20b369b07cf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:01:18.812983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1216/2425] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:01:18 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:01:21.888142 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=6 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:01:28.786081 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8d823d-41be-4d09-b388-e5ef24bac27e None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:01:28.786773 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8d823d-41be-4d09-b388-e5ef24bac27e None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:01:28.786773 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8d823d-41be-4d09-b388-e5ef24bac27e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:01:28.806395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=31 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:01:28.807049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-9b8d823d-41be-4d09-b388-e5ef24bac27e None None] MFA Rules not processed for user `54d4e437036640b58fda503c4d213f5c`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:01:28.808006 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-9b8d823d-41be-4d09-b388-e5ef24bac27e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:01:28.874150 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1210/2426] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:01:28 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:01:38.865339 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=25 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:01:48.280735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b14fcc-94e5-4ea6-8356-2e0011c15881 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:01:48.281238 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b14fcc-94e5-4ea6-8356-2e0011c15881 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:01:48.281334 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b14fcc-94e5-4ea6-8356-2e0011c15881 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:01:48.302582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-b9b14fcc-94e5-4ea6-8356-2e0011c15881 None None] MFA Rules not processed for user `294ae98a8492468893beb3b8c2870d5e`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:01:48.303415 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-b9b14fcc-94e5-4ea6-8356-2e0011c15881 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:01:48.361103 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1217/2427] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:01:48 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:01:49.149419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8d823d-41be-4d09-b388-e5ef24bac27e None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:01:49.159573 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8d823d-41be-4d09-b388-e5ef24bac27e None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:01:49.163415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522b2163-441e-4f20-9d66-1969efdfc58c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:01:49.164582 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522b2163-441e-4f20-9d66-1969efdfc58c service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:01:49.164859 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522b2163-441e-4f20-9d66-1969efdfc58c service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:01:49.165130 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522b2163-441e-4f20-9d66-1969efdfc58c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:01:49.166957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-522b2163-441e-4f20-9d66-1969efdfc58c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:01:49.180627 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522b2163-441e-4f20-9d66-1969efdfc58c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:01:49.182065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522b2163-441e-4f20-9d66-1969efdfc58c service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:01:49.195115 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1211/2428] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:01:49 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:01:52.561776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b14fcc-94e5-4ea6-8356-2e0011c15881 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:01:52.571711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b14fcc-94e5-4ea6-8356-2e0011c15881 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:01:52.575476 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4beff16a-f58e-4ac2-82d5-c897f90e2517 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:01:52.576153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4beff16a-f58e-4ac2-82d5-c897f90e2517 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:01:52.576502 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4beff16a-f58e-4ac2-82d5-c897f90e2517 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:01:52.576776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4beff16a-f58e-4ac2-82d5-c897f90e2517 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:01:52.578663 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-4beff16a-f58e-4ac2-82d5-c897f90e2517 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:01:52.593296 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4beff16a-f58e-4ac2-82d5-c897f90e2517 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:01:52.595409 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4beff16a-f58e-4ac2-82d5-c897f90e2517 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:01:52.607532 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1218/2429] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:01:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:01:59.189399 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=6 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:02.606817 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=31 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:15.791008 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522b2163-441e-4f20-9d66-1969efdfc58c service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:15.792594 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-522b2163-441e-4f20-9d66-1969efdfc58c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:15.806697 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522b2163-441e-4f20-9d66-1969efdfc58c service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:15.812482 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a993a3-b387-40c7-8914-3e37a1e60b8d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:15.813083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a993a3-b387-40c7-8914-3e37a1e60b8d service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:15.813360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a993a3-b387-40c7-8914-3e37a1e60b8d service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:15.813627 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a993a3-b387-40c7-8914-3e37a1e60b8d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:15.815342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-95a993a3-b387-40c7-8914-3e37a1e60b8d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:15.825387 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a993a3-b387-40c7-8914-3e37a1e60b8d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:15.826959 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a993a3-b387-40c7-8914-3e37a1e60b8d service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:15.835912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1212/2430] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:02:15 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:02:16.028862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4beff16a-f58e-4ac2-82d5-c897f90e2517 service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:16.031247 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-4beff16a-f58e-4ac2-82d5-c897f90e2517 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:16.102656 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4beff16a-f58e-4ac2-82d5-c897f90e2517 service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:16.102656 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1b31ac-4588-4fdf-a038-859fe402d3ce service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:16.102656 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1b31ac-4588-4fdf-a038-859fe402d3ce service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:16.102656 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1b31ac-4588-4fdf-a038-859fe402d3ce service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:16.103053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1b31ac-4588-4fdf-a038-859fe402d3ce service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:16.104589 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3b1b31ac-4588-4fdf-a038-859fe402d3ce service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:16.118630 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1b31ac-4588-4fdf-a038-859fe402d3ce service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:16.120106 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1b31ac-4588-4fdf-a038-859fe402d3ce service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:16.131022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1219/2431] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:02:16 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:02:25.831944 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=6 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:26.126906 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=26 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:30.522984 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a993a3-b387-40c7-8914-3e37a1e60b8d service nova] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:30.540014 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a993a3-b387-40c7-8914-3e37a1e60b8d service nova] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:30.546274 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c486b6-e3a0-45c0-b6e3-b33e4c0db06a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:30.546985 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c486b6-e3a0-45c0-b6e3-b33e4c0db06a service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:30.547451 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c486b6-e3a0-45c0-b6e3-b33e4c0db06a service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:30.547719 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c486b6-e3a0-45c0-b6e3-b33e4c0db06a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:30.549660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c1c486b6-e3a0-45c0-b6e3-b33e4c0db06a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:30.566360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c486b6-e3a0-45c0-b6e3-b33e4c0db06a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:30.568160 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c486b6-e3a0-45c0-b6e3-b33e4c0db06a service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:30.580637 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1213/2432] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:02:30 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:02:39.152539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1b31ac-4588-4fdf-a038-859fe402d3ce service cinder] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:39.166204 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1b31ac-4588-4fdf-a038-859fe402d3ce service cinder] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:39.172855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388f934e-8b62-42d8-9bbf-7a4d7375475b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:39.173640 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388f934e-8b62-42d8-9bbf-7a4d7375475b service nova] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:39.174021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388f934e-8b62-42d8-9bbf-7a4d7375475b service nova] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:39.174414 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388f934e-8b62-42d8-9bbf-7a4d7375475b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:39.176492 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-388f934e-8b62-42d8-9bbf-7a4d7375475b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:39.190282 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388f934e-8b62-42d8-9bbf-7a4d7375475b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:39.192779 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388f934e-8b62-42d8-9bbf-7a4d7375475b service nova] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:39.208000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1220/2433] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:02:39 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:02:39.902102 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c486b6-e3a0-45c0-b6e3-b33e4c0db06a service nova] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:39.911997 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c486b6-e3a0-45c0-b6e3-b33e4c0db06a service nova] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:39.915791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09eecfb8-ab19-40b3-b819-663911709294 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:39.916418 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09eecfb8-ab19-40b3-b819-663911709294 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:39.916698 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09eecfb8-ab19-40b3-b819-663911709294 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:39.916975 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09eecfb8-ab19-40b3-b819-663911709294 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:39.918894 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-09eecfb8-ab19-40b3-b819-663911709294 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:39.986933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09eecfb8-ab19-40b3-b819-663911709294 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:39.988827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09eecfb8-ab19-40b3-b819-663911709294 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:39.999717 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1214/2434] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:02:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:02:42.425873 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85a39f2-2201-49c5-8fb9-aedf83aa54ef None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:42.426399 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85a39f2-2201-49c5-8fb9-aedf83aa54ef None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:42.426399 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85a39f2-2201-49c5-8fb9-aedf83aa54ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:42.445933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-c85a39f2-2201-49c5-8fb9-aedf83aa54ef None None] MFA Rules not processed for user `2f9751adf8d74ab59b6b0a5d00221405`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:02:42.447270 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c85a39f2-2201-49c5-8fb9-aedf83aa54ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:42.454862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1221/2435] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:02:42 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:02:42.980807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09eecfb8-ab19-40b3-b819-663911709294 service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:42.990483 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09eecfb8-ab19-40b3-b819-663911709294 service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:42.995406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b88e4d9-6ccb-412e-b56e-f8ffeabfe9dd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:42.996021 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b88e4d9-6ccb-412e-b56e-f8ffeabfe9dd service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:42.996296 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b88e4d9-6ccb-412e-b56e-f8ffeabfe9dd service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:42.996570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b88e4d9-6ccb-412e-b56e-f8ffeabfe9dd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:42.998339 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-5b88e4d9-6ccb-412e-b56e-f8ffeabfe9dd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:43.010006 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b88e4d9-6ccb-412e-b56e-f8ffeabfe9dd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:43.011718 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b88e4d9-6ccb-412e-b56e-f8ffeabfe9dd service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:43.031265 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1215/2436] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:02:42 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:02:43.446058 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85a39f2-2201-49c5-8fb9-aedf83aa54ef None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:43.447485 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c85a39f2-2201-49c5-8fb9-aedf83aa54ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:43.465615 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85a39f2-2201-49c5-8fb9-aedf83aa54ef None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:43.471922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14297b73-5520-4bfb-a32f-800f25a3cdcf service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8f30116fb5ce43dab69efa8bafd9d049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:43.472855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14297b73-5520-4bfb-a32f-800f25a3cdcf service placement] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:43.473303 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14297b73-5520-4bfb-a32f-800f25a3cdcf service placement] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:43.473570 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14297b73-5520-4bfb-a32f-800f25a3cdcf service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:43.476046 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-14297b73-5520-4bfb-a32f-800f25a3cdcf service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:43.491143 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14297b73-5520-4bfb-a32f-800f25a3cdcf service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:43.493211 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14297b73-5520-4bfb-a32f-800f25a3cdcf service placement] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:43.515876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1222/2437] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:02:43 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:02:43.628289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b88e4d9-6ccb-412e-b56e-f8ffeabfe9dd service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:43.632529 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-5b88e4d9-6ccb-412e-b56e-f8ffeabfe9dd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:43.650430 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b88e4d9-6ccb-412e-b56e-f8ffeabfe9dd service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:43.654525 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d893df4e-57d1-476f-ae53-abfca4eedf52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:43.655168 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d893df4e-57d1-476f-ae53-abfca4eedf52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:43.655588 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d893df4e-57d1-476f-ae53-abfca4eedf52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:43.656019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d893df4e-57d1-476f-ae53-abfca4eedf52 admin admin] PATH_INFO: `/v3/users/54d4e437036640b58fda503c4d213f5c` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:43.657165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d893df4e-57d1-476f-ae53-abfca4eedf52 admin admin] RBAC: Authorizing `identity:delete_user(user_id=54d4e437036640b58fda503c4d213f5c)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:43.659354 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d893df4e-57d1-476f-ae53-abfca4eedf52 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:43.724943 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-d893df4e-57d1-476f-ae53-abfca4eedf52 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '54d4e437036640b58fda503c4d213f5c'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:43.735617 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-d893df4e-57d1-476f-ae53-abfca4eedf52 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '54d4e437036640b58fda503c4d213f5c'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:43.736225 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-d893df4e-57d1-476f-ae53-abfca4eedf52 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '54d4e437036640b58fda503c4d213f5c'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:43.736825 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-d893df4e-57d1-476f-ae53-abfca4eedf52 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '54d4e437036640b58fda503c4d213f5c'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:43.745182 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-d893df4e-57d1-476f-ae53-abfca4eedf52 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '54d4e437036640b58fda503c4d213f5c'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:43.773011 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1216/2438] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:02:43 2022] DELETE /identity/v3/users/54d4e437036640b58fda503c4d213f5c => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:02:43.969765 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14297b73-5520-4bfb-a32f-800f25a3cdcf service placement] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:43.984237 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14297b73-5520-4bfb-a32f-800f25a3cdcf service placement] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:44.078953 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3925e5f-979f-4d38-8d66-c791776bb916 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:44.079667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3925e5f-979f-4d38-8d66-c791776bb916 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:44.080033 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3925e5f-979f-4d38-8d66-c791776bb916 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:44.080225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3925e5f-979f-4d38-8d66-c791776bb916 admin admin] PATH_INFO: `/v3/projects/df01daf86854497fad1d1d016c269ef1` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:44.081662 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3925e5f-979f-4d38-8d66-c791776bb916 admin admin] RBAC: Authorizing `identity:delete_project(project_id=df01daf86854497fad1d1d016c269ef1)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:44.083798 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3925e5f-979f-4d38-8d66-c791776bb916 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:44.133666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a3925e5f-979f-4d38-8d66-c791776bb916 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'df01daf86854497fad1d1d016c269ef1', 'request_id': 'req-a3925e5f-979f-4d38-8d66-c791776bb916'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:44.139594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a3925e5f-979f-4d38-8d66-c791776bb916 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'df01daf86854497fad1d1d016c269ef1', 'request_id': 'req-a3925e5f-979f-4d38-8d66-c791776bb916'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:44.148268 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a3925e5f-979f-4d38-8d66-c791776bb916 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'df01daf86854497fad1d1d016c269ef1', 'request_id': 'req-a3925e5f-979f-4d38-8d66-c791776bb916'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:44.160716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a3925e5f-979f-4d38-8d66-c791776bb916 admin admin] The token cache is being invalidate because project df01daf86854497fad1d1d016c269ef1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:02:44.161010 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a3925e5f-979f-4d38-8d66-c791776bb916 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:44.161681 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a3925e5f-979f-4d38-8d66-c791776bb916 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:44.163311 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1223/2439] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:02:43 2022] DELETE /identity/v3/projects/df01daf86854497fad1d1d016c269ef1 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:02:44.168374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb477e23-9378-43c2-91a8-e52b7e498281 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:44.168959 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb477e23-9378-43c2-91a8-e52b7e498281 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:44.168959 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb477e23-9378-43c2-91a8-e52b7e498281 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:44.190729 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-bb477e23-9378-43c2-91a8-e52b7e498281 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:02:44.191941 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-bb477e23-9378-43c2-91a8-e52b7e498281 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:44.229679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:44.262098 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1217/2440] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:02:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 01:02:44.269628 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95f9a47-ebd8-4c98-9a14-f31517b237da None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:44.269924 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95f9a47-ebd8-4c98-9a14-f31517b237da None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:44.270208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95f9a47-ebd8-4c98-9a14-f31517b237da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:44.295927 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-a95f9a47-ebd8-4c98-9a14-f31517b237da None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:02:44.297697 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a95f9a47-ebd8-4c98-9a14-f31517b237da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:44.307325 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1224/2441] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:02:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:02:44.312741 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb477e23-9378-43c2-91a8-e52b7e498281 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:44.325818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb477e23-9378-43c2-91a8-e52b7e498281 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:44.331524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785fc323-048d-43b9-a26b-04c3f926c278 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:44.332159 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785fc323-048d-43b9-a26b-04c3f926c278 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:44.332650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785fc323-048d-43b9-a26b-04c3f926c278 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:44.333328 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785fc323-048d-43b9-a26b-04c3f926c278 admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:44.334191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785fc323-048d-43b9-a26b-04c3f926c278 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:02:44.334528 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785fc323-048d-43b9-a26b-04c3f926c278 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:44.336440 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785fc323-048d-43b9-a26b-04c3f926c278 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:44.352832 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1218/2442] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:02:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 01:02:44.356925 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95f9a47-ebd8-4c98-9a14-f31517b237da None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:44.367185 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95f9a47-ebd8-4c98-9a14-f31517b237da None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:44.374632 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457ab87a-c188-43e0-a584-f770d5096ee6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:44.375463 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457ab87a-c188-43e0-a584-f770d5096ee6 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:44.375910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457ab87a-c188-43e0-a584-f770d5096ee6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:44.376190 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457ab87a-c188-43e0-a584-f770d5096ee6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:44.376855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457ab87a-c188-43e0-a584-f770d5096ee6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:44.378835 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457ab87a-c188-43e0-a584-f770d5096ee6 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:44.419185 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1225/2443] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:02:44 2022] POST /identity/v3/projects => generated 375 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:02:44.423903 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785fc323-048d-43b9-a26b-04c3f926c278 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:44.442274 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785fc323-048d-43b9-a26b-04c3f926c278 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:44.505502 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1924433a-77d6-4ccb-b877-6f83ea2064fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:44.506656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1924433a-77d6-4ccb-b877-6f83ea2064fb admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:44.506656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1924433a-77d6-4ccb-b877-6f83ea2064fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:44.506656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1924433a-77d6-4ccb-b877-6f83ea2064fb admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:44.507306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1924433a-77d6-4ccb-b877-6f83ea2064fb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:44.509286 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1924433a-77d6-4ccb-b877-6f83ea2064fb admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:44.542066 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1219/2444] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:02:44 2022] POST /identity/v3/users => generated 288 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:02:44.543179 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457ab87a-c188-43e0-a584-f770d5096ee6 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:44.553830 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457ab87a-c188-43e0-a584-f770d5096ee6 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:44.559739 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd2aaf0-5665-4b8c-8fe8-e0cdf11033eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:44.560717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd2aaf0-5665-4b8c-8fe8-e0cdf11033eb admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:44.560996 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd2aaf0-5665-4b8c-8fe8-e0cdf11033eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:44.561348 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd2aaf0-5665-4b8c-8fe8-e0cdf11033eb admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:44.561912 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd2aaf0-5665-4b8c-8fe8-e0cdf11033eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:44.563573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd2aaf0-5665-4b8c-8fe8-e0cdf11033eb admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:44.577764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1226/2445] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:02:44 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 01:02:44.579005 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1924433a-77d6-4ccb-b877-6f83ea2064fb admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:44.590004 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1924433a-77d6-4ccb-b877-6f83ea2064fb admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:44.594953 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f3764d-a751-4ef5-9020-94e1bfaa28d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:44.596039 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f3764d-a751-4ef5-9020-94e1bfaa28d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:44.596039 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f3764d-a751-4ef5-9020-94e1bfaa28d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:44.596290 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f3764d-a751-4ef5-9020-94e1bfaa28d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:44.596767 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f3764d-a751-4ef5-9020-94e1bfaa28d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:44.598355 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f3764d-a751-4ef5-9020-94e1bfaa28d8 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:44.609913 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1220/2446] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:02:44 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:02:44.614115 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd2aaf0-5665-4b8c-8fe8-e0cdf11033eb admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:44.628519 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd2aaf0-5665-4b8c-8fe8-e0cdf11033eb admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:44.633732 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a20cc68-a983-48e0-8700-ffaaba2a323d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:44.634666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a20cc68-a983-48e0-8700-ffaaba2a323d admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:44.635115 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a20cc68-a983-48e0-8700-ffaaba2a323d admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:44.635545 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a20cc68-a983-48e0-8700-ffaaba2a323d admin admin] PATH_INFO: `/v3/projects/250267bfa41e4620bf154ec96fcf64ae/users/151d35e52e6b4a3e83607cc70c2b396c/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:44.652874 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a20cc68-a983-48e0-8700-ffaaba2a323d admin admin] RBAC: Authorizing `identity:create_grant(project_id=250267bfa41e4620bf154ec96fcf64ae, user_id=151d35e52e6b4a3e83607cc70c2b396c, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:44.656036 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a20cc68-a983-48e0-8700-ffaaba2a323d admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:44.658447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:44.673137 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1227/2447] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:02:44 2022] PUT /identity/v3/projects/250267bfa41e4620bf154ec96fcf64ae/users/151d35e52e6b4a3e83607cc70c2b396c/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:02:44.684632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93bae77-6f14-4c0e-a08a-a14b07a9a36e None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:44.684899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93bae77-6f14-4c0e-a08a-a14b07a9a36e None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:44.685220 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93bae77-6f14-4c0e-a08a-a14b07a9a36e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:44.721153 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-e93bae77-6f14-4c0e-a08a-a14b07a9a36e None None] MFA Rules not processed for user `151d35e52e6b4a3e83607cc70c2b396c`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:02:44.722392 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-e93bae77-6f14-4c0e-a08a-a14b07a9a36e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:44.794808 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1221/2448] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:02:44 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:02:44.814591 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a20cc68-a983-48e0-8700-ffaaba2a323d admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:44.816487 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-5a20cc68-a983-48e0-8700-ffaaba2a323d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:44.910681 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a20cc68-a983-48e0-8700-ffaaba2a323d admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:44.923298 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9477a68-7c2d-49ee-83bd-251f71475732 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:44.923298 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9477a68-7c2d-49ee-83bd-251f71475732 service nova] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:44.923298 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9477a68-7c2d-49ee-83bd-251f71475732 service nova] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:44.923298 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9477a68-7c2d-49ee-83bd-251f71475732 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:44.937638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9477a68-7c2d-49ee-83bd-251f71475732 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:44.942581 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9477a68-7c2d-49ee-83bd-251f71475732 service nova] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:44.953804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1228/2449] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:02:44 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:02:45.248222 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93bae77-6f14-4c0e-a08a-a14b07a9a36e None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:45.248750 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9477a68-7c2d-49ee-83bd-251f71475732 service nova] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:45.249737 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a9477a68-7c2d-49ee-83bd-251f71475732 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:45.341811 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:45.345431 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9477a68-7c2d-49ee-83bd-251f71475732 service nova] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:45.353073 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644838d7-f979-4f72-92c3-0ff6912b6319 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:45.356011 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644838d7-f979-4f72-92c3-0ff6912b6319 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:45.356633 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644838d7-f979-4f72-92c3-0ff6912b6319 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:45.357168 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644838d7-f979-4f72-92c3-0ff6912b6319 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:45.361772 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-644838d7-f979-4f72-92c3-0ff6912b6319 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:45.384224 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644838d7-f979-4f72-92c3-0ff6912b6319 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:45.387541 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644838d7-f979-4f72-92c3-0ff6912b6319 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:45.399648 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93bae77-6f14-4c0e-a08a-a14b07a9a36e None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:45.401787 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1229/2450] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:02:45 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:02:45.413264 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d33d45-093b-4634-a926-54d2369769a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:45.414490 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d33d45-093b-4634-a926-54d2369769a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:45.414650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d33d45-093b-4634-a926-54d2369769a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:45.415097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d33d45-093b-4634-a926-54d2369769a5 admin admin] PATH_INFO: `/v3/users/151d35e52e6b4a3e83607cc70c2b396c` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:45.418412 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d33d45-093b-4634-a926-54d2369769a5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=151d35e52e6b4a3e83607cc70c2b396c)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:45.420690 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d33d45-093b-4634-a926-54d2369769a5 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:45.496597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-62d33d45-093b-4634-a926-54d2369769a5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '151d35e52e6b4a3e83607cc70c2b396c'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:45.506018 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-62d33d45-093b-4634-a926-54d2369769a5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '151d35e52e6b4a3e83607cc70c2b396c'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:45.507272 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-62d33d45-093b-4634-a926-54d2369769a5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '151d35e52e6b4a3e83607cc70c2b396c'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:45.508962 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-62d33d45-093b-4634-a926-54d2369769a5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '151d35e52e6b4a3e83607cc70c2b396c'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:45.521071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-62d33d45-093b-4634-a926-54d2369769a5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '151d35e52e6b4a3e83607cc70c2b396c'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:45.542355 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1222/2451] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:02:45 2022] DELETE /identity/v3/users/151d35e52e6b4a3e83607cc70c2b396c => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:02:45.547405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24890fc9-9a4c-493b-a187-8954a19e461d None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:45.548444 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24890fc9-9a4c-493b-a187-8954a19e461d None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:45.549348 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24890fc9-9a4c-493b-a187-8954a19e461d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:45.573583 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-24890fc9-9a4c-493b-a187-8954a19e461d None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:02:45.575929 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-24890fc9-9a4c-493b-a187-8954a19e461d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:45.662553 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1230/2452] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:02:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 01:02:45.675147 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d33d45-093b-4634-a926-54d2369769a5 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:45.676711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-62d33d45-093b-4634-a926-54d2369769a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:45.752823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d33d45-093b-4634-a926-54d2369769a5 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:45.757154 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e881f5-402e-4821-9d4b-47d5be306097 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:45.758426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e881f5-402e-4821-9d4b-47d5be306097 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:45.758828 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e881f5-402e-4821-9d4b-47d5be306097 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:45.759199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e881f5-402e-4821-9d4b-47d5be306097 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:45.770636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e881f5-402e-4821-9d4b-47d5be306097 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:45.772215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e881f5-402e-4821-9d4b-47d5be306097 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:45.783775 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1223/2453] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:02:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:02:46.155592 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24890fc9-9a4c-493b-a187-8954a19e461d None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:46.156998 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-24890fc9-9a4c-493b-a187-8954a19e461d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:46.205959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:46.277404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24890fc9-9a4c-493b-a187-8954a19e461d None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:46.283911 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5075e661-8047-4612-8a73-97fe19639995 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:46.284652 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5075e661-8047-4612-8a73-97fe19639995 service nova] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:46.285034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5075e661-8047-4612-8a73-97fe19639995 service nova] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:46.285407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5075e661-8047-4612-8a73-97fe19639995 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:46.287345 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-5075e661-8047-4612-8a73-97fe19639995 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:46.362703 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5075e661-8047-4612-8a73-97fe19639995 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:46.364432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5075e661-8047-4612-8a73-97fe19639995 service nova] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:46.376520 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1231/2454] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:02:46 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 222 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:02:46.390128 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e881f5-402e-4821-9d4b-47d5be306097 service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:46.392772 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-26e881f5-402e-4821-9d4b-47d5be306097 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:46.412629 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e881f5-402e-4821-9d4b-47d5be306097 service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:46.420011 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf73a28-55b3-4d10-bb58-9cb9e51badda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:46.421219 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf73a28-55b3-4d10-bb58-9cb9e51badda admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:46.421707 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf73a28-55b3-4d10-bb58-9cb9e51badda admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:46.422411 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf73a28-55b3-4d10-bb58-9cb9e51badda admin admin] PATH_INFO: `/v3/projects/250267bfa41e4620bf154ec96fcf64ae` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:46.424019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf73a28-55b3-4d10-bb58-9cb9e51badda admin admin] RBAC: Authorizing `identity:delete_project(project_id=250267bfa41e4620bf154ec96fcf64ae)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:46.426133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf73a28-55b3-4d10-bb58-9cb9e51badda admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:46.499900 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-edf73a28-55b3-4d10-bb58-9cb9e51badda admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '250267bfa41e4620bf154ec96fcf64ae', 'request_id': 'req-edf73a28-55b3-4d10-bb58-9cb9e51badda'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:46.503296 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:46.529712 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-edf73a28-55b3-4d10-bb58-9cb9e51badda admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '250267bfa41e4620bf154ec96fcf64ae', 'request_id': 'req-edf73a28-55b3-4d10-bb58-9cb9e51badda'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:46.555533 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-edf73a28-55b3-4d10-bb58-9cb9e51badda admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '250267bfa41e4620bf154ec96fcf64ae', 'request_id': 'req-edf73a28-55b3-4d10-bb58-9cb9e51badda'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:46.579840 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-edf73a28-55b3-4d10-bb58-9cb9e51badda admin admin] The token cache is being invalidate because project 250267bfa41e4620bf154ec96fcf64ae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:02:46.579986 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-edf73a28-55b3-4d10-bb58-9cb9e51badda admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:46.581075 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-edf73a28-55b3-4d10-bb58-9cb9e51badda admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:46.583130 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1224/2455] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:02:46 2022] DELETE /identity/v3/projects/250267bfa41e4620bf154ec96fcf64ae => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:02:46.588474 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69b5097-44bb-44cf-98c4-15083b33d661 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:46.588977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69b5097-44bb-44cf-98c4-15083b33d661 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:46.589090 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69b5097-44bb-44cf-98c4-15083b33d661 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:46.612681 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-b69b5097-44bb-44cf-98c4-15083b33d661 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:02:46.613920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-b69b5097-44bb-44cf-98c4-15083b33d661 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:46.687542 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1232/2456] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:02:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:02:46.695770 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c85415-300c-4122-aec6-2f32c35d4a5f None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:46.696109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c85415-300c-4122-aec6-2f32c35d4a5f None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:46.696318 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c85415-300c-4122-aec6-2f32c35d4a5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:46.719483 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-48c85415-300c-4122-aec6-2f32c35d4a5f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:02:46.720872 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-48c85415-300c-4122-aec6-2f32c35d4a5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:46.735560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1225/2457] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:02:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:02:46.738965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69b5097-44bb-44cf-98c4-15083b33d661 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:46.751817 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69b5097-44bb-44cf-98c4-15083b33d661 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:46.758622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc1430f-b1f1-41d3-aee1-77b537acd608 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:46.759346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc1430f-b1f1-41d3-aee1-77b537acd608 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:46.759732 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc1430f-b1f1-41d3-aee1-77b537acd608 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:46.760076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc1430f-b1f1-41d3-aee1-77b537acd608 admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:46.760918 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc1430f-b1f1-41d3-aee1-77b537acd608 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:02:46.761288 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc1430f-b1f1-41d3-aee1-77b537acd608 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:46.763881 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc1430f-b1f1-41d3-aee1-77b537acd608 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:46.779316 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1233/2458] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:02:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 01:02:46.783263 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c85415-300c-4122-aec6-2f32c35d4a5f None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:46.801754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c85415-300c-4122-aec6-2f32c35d4a5f None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:46.809709 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8abfbb-06c0-40b7-96ef-6092ad4f0c5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:46.810969 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8abfbb-06c0-40b7-96ef-6092ad4f0c5e admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:46.811598 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8abfbb-06c0-40b7-96ef-6092ad4f0c5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:46.811886 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8abfbb-06c0-40b7-96ef-6092ad4f0c5e admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:46.812570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8abfbb-06c0-40b7-96ef-6092ad4f0c5e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:46.814622 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8abfbb-06c0-40b7-96ef-6092ad4f0c5e admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:46.864667 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1226/2459] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:02:46 2022] POST /identity/v3/projects => generated 381 bytes in 82 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:02:46.868408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc1430f-b1f1-41d3-aee1-77b537acd608 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:46.890499 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc1430f-b1f1-41d3-aee1-77b537acd608 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:46.973372 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:47.003160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c48c664-1c09-4c81-86cc-39b9d0733ced admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:47.003882 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c48c664-1c09-4c81-86cc-39b9d0733ced admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:47.004167 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c48c664-1c09-4c81-86cc-39b9d0733ced admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:47.004487 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c48c664-1c09-4c81-86cc-39b9d0733ced admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:47.005173 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c48c664-1c09-4c81-86cc-39b9d0733ced admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:47.007247 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c48c664-1c09-4c81-86cc-39b9d0733ced admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:47.042403 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1234/2460] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:02:46 2022] POST /identity/v3/users => generated 291 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:02:47.047310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8abfbb-06c0-40b7-96ef-6092ad4f0c5e admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:47.061755 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8abfbb-06c0-40b7-96ef-6092ad4f0c5e admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:47.076422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b7fcee-fdb8-4e59-9aac-4c7efbf62a5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:47.077464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b7fcee-fdb8-4e59-9aac-4c7efbf62a5a admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:47.077464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b7fcee-fdb8-4e59-9aac-4c7efbf62a5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:47.077688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b7fcee-fdb8-4e59-9aac-4c7efbf62a5a admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:47.078906 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b7fcee-fdb8-4e59-9aac-4c7efbf62a5a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:47.080089 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b7fcee-fdb8-4e59-9aac-4c7efbf62a5a admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:47.100431 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1227/2461] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:02:47 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 01:02:47.107735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c48c664-1c09-4c81-86cc-39b9d0733ced admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:47.121896 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c48c664-1c09-4c81-86cc-39b9d0733ced admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:47.131060 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16d79e0-6245-40f8-b44f-15e22368e682 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:47.132094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16d79e0-6245-40f8-b44f-15e22368e682 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:47.133857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16d79e0-6245-40f8-b44f-15e22368e682 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:47.134302 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16d79e0-6245-40f8-b44f-15e22368e682 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:47.135171 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a16d79e0-6245-40f8-b44f-15e22368e682 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:47.137029 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a16d79e0-6245-40f8-b44f-15e22368e682 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:47.157972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1235/2462] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:02:47 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:02:47.162123 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b7fcee-fdb8-4e59-9aac-4c7efbf62a5a admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:47.177899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b7fcee-fdb8-4e59-9aac-4c7efbf62a5a admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:47.186150 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d246b6-cab0-4330-82a8-c641a318cac9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:47.187174 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d246b6-cab0-4330-82a8-c641a318cac9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:47.187558 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d246b6-cab0-4330-82a8-c641a318cac9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:47.187979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d246b6-cab0-4330-82a8-c641a318cac9 admin admin] PATH_INFO: `/v3/projects/f877d8f0958f4952b607685b1a5cb618/users/d6be984a1597429db6cf0f6d10b3a6ff/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:47.218480 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d246b6-cab0-4330-82a8-c641a318cac9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f877d8f0958f4952b607685b1a5cb618, user_id=d6be984a1597429db6cf0f6d10b3a6ff, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:47.222764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d246b6-cab0-4330-82a8-c641a318cac9 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:47.244895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1228/2463] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:02:47 2022] PUT /identity/v3/projects/f877d8f0958f4952b607685b1a5cb618/users/d6be984a1597429db6cf0f6d10b3a6ff/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:02:47.255326 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5597f6aa-2a10-4692-89ee-721c6a555231 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:47.255620 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5597f6aa-2a10-4692-89ee-721c6a555231 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:47.255899 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5597f6aa-2a10-4692-89ee-721c6a555231 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:47.336301 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-5597f6aa-2a10-4692-89ee-721c6a555231 None None] MFA Rules not processed for user `d6be984a1597429db6cf0f6d10b3a6ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:02:47.337750 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-5597f6aa-2a10-4692-89ee-721c6a555231 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:47.461321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1236/2464] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:02:47 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:02:47.464946 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d246b6-cab0-4330-82a8-c641a318cac9 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:47.555279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d246b6-cab0-4330-82a8-c641a318cac9 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:47.559955 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a33dc1-a9b4-4614-9e06-15f4cb5ce48e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:47.560563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a33dc1-a9b4-4614-9e06-15f4cb5ce48e admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:47.560837 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a33dc1-a9b4-4614-9e06-15f4cb5ce48e admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:47.561109 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a33dc1-a9b4-4614-9e06-15f4cb5ce48e admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:47.561743 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a33dc1-a9b4-4614-9e06-15f4cb5ce48e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:47.563654 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a33dc1-a9b4-4614-9e06-15f4cb5ce48e admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:47.596956 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1229/2465] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:02:47 2022] POST /identity/v3/projects => generated 377 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:02:47.598077 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5597f6aa-2a10-4692-89ee-721c6a555231 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:47.609076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5597f6aa-2a10-4692-89ee-721c6a555231 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:47.667089 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cac5c88-008a-426e-8fa3-a59e19eddb1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:47.667711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cac5c88-008a-426e-8fa3-a59e19eddb1c admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:47.667996 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cac5c88-008a-426e-8fa3-a59e19eddb1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:47.668299 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cac5c88-008a-426e-8fa3-a59e19eddb1c admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:47.668948 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cac5c88-008a-426e-8fa3-a59e19eddb1c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:47.670949 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cac5c88-008a-426e-8fa3-a59e19eddb1c admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:47.695276 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1237/2466] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:02:47 2022] POST /identity/v3/users => generated 295 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:02:47.700094 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a33dc1-a9b4-4614-9e06-15f4cb5ce48e admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:47.711031 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a33dc1-a9b4-4614-9e06-15f4cb5ce48e admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:47.715796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57b4c4a-3759-40cf-8dbe-89017c3b902b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:47.716405 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57b4c4a-3759-40cf-8dbe-89017c3b902b admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:47.716906 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57b4c4a-3759-40cf-8dbe-89017c3b902b admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:47.717049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57b4c4a-3759-40cf-8dbe-89017c3b902b admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:47.717634 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57b4c4a-3759-40cf-8dbe-89017c3b902b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:47.719391 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57b4c4a-3759-40cf-8dbe-89017c3b902b admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:47.730718 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1230/2467] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:02:47 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:02:47.734840 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cac5c88-008a-426e-8fa3-a59e19eddb1c admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:47.746996 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cac5c88-008a-426e-8fa3-a59e19eddb1c admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:47.752115 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ad5532-484b-4e0a-8fd5-6a01a50a4aaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:47.752800 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ad5532-484b-4e0a-8fd5-6a01a50a4aaa admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:47.753110 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ad5532-484b-4e0a-8fd5-6a01a50a4aaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:47.753409 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ad5532-484b-4e0a-8fd5-6a01a50a4aaa admin admin] PATH_INFO: `/v3/projects/8e5dbf82811144e7b8b7cefaf241f4ce/users/58f9818c16f443ffabb285d39a5105be/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:47.772859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ad5532-484b-4e0a-8fd5-6a01a50a4aaa admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e5dbf82811144e7b8b7cefaf241f4ce, user_id=58f9818c16f443ffabb285d39a5105be, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:47.775715 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ad5532-484b-4e0a-8fd5-6a01a50a4aaa admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:47.792880 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1238/2468] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:02:47 2022] PUT /identity/v3/projects/8e5dbf82811144e7b8b7cefaf241f4ce/users/58f9818c16f443ffabb285d39a5105be/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 01:02:47.798889 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57b4c4a-3759-40cf-8dbe-89017c3b902b admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:47.812956 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57b4c4a-3759-40cf-8dbe-89017c3b902b admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:47.872262 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:47.887818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5201f1ce-a46b-431e-99fa-7f373b7b564a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:47.888596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5201f1ce-a46b-431e-99fa-7f373b7b564a admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:47.888955 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5201f1ce-a46b-431e-99fa-7f373b7b564a admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:47.889461 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5201f1ce-a46b-431e-99fa-7f373b7b564a admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:47.890505 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5201f1ce-a46b-431e-99fa-7f373b7b564a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:47.892578 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5201f1ce-a46b-431e-99fa-7f373b7b564a admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:47.907070 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1231/2469] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:02:47 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:02:47.910652 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ad5532-484b-4e0a-8fd5-6a01a50a4aaa admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:47.922252 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ad5532-484b-4e0a-8fd5-6a01a50a4aaa admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:47.928173 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30d4ee8-31be-48e9-87b8-af2470e5821f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:47.928970 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30d4ee8-31be-48e9-87b8-af2470e5821f admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:47.929264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30d4ee8-31be-48e9-87b8-af2470e5821f admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:47.929638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30d4ee8-31be-48e9-87b8-af2470e5821f admin admin] PATH_INFO: `/v3/projects/8e5dbf82811144e7b8b7cefaf241f4ce/users/58f9818c16f443ffabb285d39a5105be/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:47.932342 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30d4ee8-31be-48e9-87b8-af2470e5821f admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e5dbf82811144e7b8b7cefaf241f4ce, user_id=58f9818c16f443ffabb285d39a5105be, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:47.935056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30d4ee8-31be-48e9-87b8-af2470e5821f admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:47.954566 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1239/2470] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:02:47 2022] PUT /identity/v3/projects/8e5dbf82811144e7b8b7cefaf241f4ce/users/58f9818c16f443ffabb285d39a5105be/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:02:47.963133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62aa2de0-32ad-4775-91db-fee45020ca89 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:47.963419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62aa2de0-32ad-4775-91db-fee45020ca89 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:47.963707 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62aa2de0-32ad-4775-91db-fee45020ca89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:48.027943 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-62aa2de0-32ad-4775-91db-fee45020ca89 None None] MFA Rules not processed for user `58f9818c16f443ffabb285d39a5105be`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:02:48.027943 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-62aa2de0-32ad-4775-91db-fee45020ca89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:48.159596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1232/2471] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:02:47 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:02:48.173716 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30d4ee8-31be-48e9-87b8-af2470e5821f admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:48.177151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-e30d4ee8-31be-48e9-87b8-af2470e5821f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:48.193923 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:48.298625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30d4ee8-31be-48e9-87b8-af2470e5821f admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:48.303330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1572be8e-7a83-4e0d-9ce3-b5ca8860bb3c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:48.303540 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1572be8e-7a83-4e0d-9ce3-b5ca8860bb3c service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:48.303753 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1572be8e-7a83-4e0d-9ce3-b5ca8860bb3c service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:48.304010 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1572be8e-7a83-4e0d-9ce3-b5ca8860bb3c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:48.315747 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1572be8e-7a83-4e0d-9ce3-b5ca8860bb3c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:48.317236 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1572be8e-7a83-4e0d-9ce3-b5ca8860bb3c service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:48.354266 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1240/2472] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:02:48 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:02:48.859706 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62aa2de0-32ad-4775-91db-fee45020ca89 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:48.876861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62aa2de0-32ad-4775-91db-fee45020ca89 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:48.881742 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c849d86-0c9f-4759-ac62-44ccf0e5adc6 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58f9818c16f443ffabb285d39a5105be', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8e5dbf82811144e7b8b7cefaf241f4ce', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:48.882364 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c849d86-0c9f-4759-ac62-44ccf0e5adc6 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:48.882808 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c849d86-0c9f-4759-ac62-44ccf0e5adc6 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:48.883761 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c849d86-0c9f-4759-ac62-44ccf0e5adc6 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:48.884436 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c849d86-0c9f-4759-ac62-44ccf0e5adc6 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:02:48.884752 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c849d86-0c9f-4759-ac62-44ccf0e5adc6 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:48.886974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c849d86-0c9f-4759-ac62-44ccf0e5adc6 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:48.902244 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1233/2473] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:02:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 01:02:48.906229 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1572be8e-7a83-4e0d-9ce3-b5ca8860bb3c service cinder] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:48.915927 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1572be8e-7a83-4e0d-9ce3-b5ca8860bb3c service cinder] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:48.923498 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d76ebd-1aea-4713-b6a4-f09e13421b08 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58f9818c16f443ffabb285d39a5105be', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8e5dbf82811144e7b8b7cefaf241f4ce', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:48.924597 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d76ebd-1aea-4713-b6a4-f09e13421b08 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:48.925794 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d76ebd-1aea-4713-b6a4-f09e13421b08 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:48.926361 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d76ebd-1aea-4713-b6a4-f09e13421b08 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:48.927617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d76ebd-1aea-4713-b6a4-f09e13421b08 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:48.929903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d76ebd-1aea-4713-b6a4-f09e13421b08 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:48.965114 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1241/2474] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:02:48 2022] POST /identity/v3/projects => generated 354 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 01:02:48.969738 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c849d86-0c9f-4759-ac62-44ccf0e5adc6 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:48.982389 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c849d86-0c9f-4759-ac62-44ccf0e5adc6 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:49.050739 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635b7609-d4e5-4f8f-9334-7d94333d6c8e tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58f9818c16f443ffabb285d39a5105be', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8e5dbf82811144e7b8b7cefaf241f4ce', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:49.051422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635b7609-d4e5-4f8f-9334-7d94333d6c8e tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:49.051700 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635b7609-d4e5-4f8f-9334-7d94333d6c8e tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:49.051980 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635b7609-d4e5-4f8f-9334-7d94333d6c8e tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:49.052654 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635b7609-d4e5-4f8f-9334-7d94333d6c8e tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:02:49.052952 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635b7609-d4e5-4f8f-9334-7d94333d6c8e tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:49.054672 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635b7609-d4e5-4f8f-9334-7d94333d6c8e tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:49.071488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1234/2475] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:02:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 01:02:49.113449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d76ebd-1aea-4713-b6a4-f09e13421b08 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:49.130900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d76ebd-1aea-4713-b6a4-f09e13421b08 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:49.146725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad77886-82de-4f64-a3a1-b8348681e416 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58f9818c16f443ffabb285d39a5105be', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8e5dbf82811144e7b8b7cefaf241f4ce', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:49.146725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad77886-82de-4f64-a3a1-b8348681e416 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:49.146725 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad77886-82de-4f64-a3a1-b8348681e416 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:49.147014 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad77886-82de-4f64-a3a1-b8348681e416 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] PATH_INFO: `/v3/projects/db29363b95bc4b1e8025bd80562989c1` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:49.147014 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad77886-82de-4f64-a3a1-b8348681e416 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] RBAC: Authorizing `identity:delete_project(project_id=db29363b95bc4b1e8025bd80562989c1)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:49.147014 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad77886-82de-4f64-a3a1-b8348681e416 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:49.187671 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5ad77886-82de-4f64-a3a1-b8348681e416 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'db29363b95bc4b1e8025bd80562989c1', 'request_id': 'req-5ad77886-82de-4f64-a3a1-b8348681e416'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:49.193059 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5ad77886-82de-4f64-a3a1-b8348681e416 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'db29363b95bc4b1e8025bd80562989c1', 'request_id': 'req-5ad77886-82de-4f64-a3a1-b8348681e416'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:49.201360 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5ad77886-82de-4f64-a3a1-b8348681e416 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'db29363b95bc4b1e8025bd80562989c1', 'request_id': 'req-5ad77886-82de-4f64-a3a1-b8348681e416'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:49.213193 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5ad77886-82de-4f64-a3a1-b8348681e416 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] The token cache is being invalidate because project db29363b95bc4b1e8025bd80562989c1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:02:49.213512 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5ad77886-82de-4f64-a3a1-b8348681e416 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:49.214227 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5ad77886-82de-4f64-a3a1-b8348681e416 tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:49.215740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1242/2476] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:02:49 2022] DELETE /identity/v3/projects/db29363b95bc4b1e8025bd80562989c1 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 01:02:49.403025 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635b7609-d4e5-4f8f-9334-7d94333d6c8e tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:49.404797 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-635b7609-d4e5-4f8f-9334-7d94333d6c8e tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:49.457435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:49.470331 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635b7609-d4e5-4f8f-9334-7d94333d6c8e tempest-VolumeQuotaClassesTest-22251342 tempest-VolumeQuotaClassesTest-22251342-project-admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:49.525877 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa2f4c5-119b-48cc-9cdd-d90936b7c07b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:49.526545 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa2f4c5-119b-48cc-9cdd-d90936b7c07b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:49.526829 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa2f4c5-119b-48cc-9cdd-d90936b7c07b admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:49.527095 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa2f4c5-119b-48cc-9cdd-d90936b7c07b admin admin] PATH_INFO: `/v3/users/d6be984a1597429db6cf0f6d10b3a6ff` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:49.528219 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa2f4c5-119b-48cc-9cdd-d90936b7c07b admin admin] RBAC: Authorizing `identity:delete_user(user_id=d6be984a1597429db6cf0f6d10b3a6ff)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:49.530115 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa2f4c5-119b-48cc-9cdd-d90936b7c07b admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:49.576128 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-daa2f4c5-119b-48cc-9cdd-d90936b7c07b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd6be984a1597429db6cf0f6d10b3a6ff'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:49.582987 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-daa2f4c5-119b-48cc-9cdd-d90936b7c07b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd6be984a1597429db6cf0f6d10b3a6ff'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:49.583555 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-daa2f4c5-119b-48cc-9cdd-d90936b7c07b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd6be984a1597429db6cf0f6d10b3a6ff'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:49.584165 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-daa2f4c5-119b-48cc-9cdd-d90936b7c07b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd6be984a1597429db6cf0f6d10b3a6ff'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:49.592007 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-daa2f4c5-119b-48cc-9cdd-d90936b7c07b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd6be984a1597429db6cf0f6d10b3a6ff'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:49.609284 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1235/2477] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:02:49 2022] DELETE /identity/v3/users/d6be984a1597429db6cf0f6d10b3a6ff => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:02:49.613703 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939fd1d2-f71b-41a2-a9e3-8d456464c622 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:49.613989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939fd1d2-f71b-41a2-a9e3-8d456464c622 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:49.614265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939fd1d2-f71b-41a2-a9e3-8d456464c622 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:49.632557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-939fd1d2-f71b-41a2-a9e3-8d456464c622 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:02:49.634241 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-939fd1d2-f71b-41a2-a9e3-8d456464c622 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:49.691321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1243/2478] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:02:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:02:49.701399 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa2f4c5-119b-48cc-9cdd-d90936b7c07b admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:49.702857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-daa2f4c5-119b-48cc-9cdd-d90936b7c07b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:49.758825 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa2f4c5-119b-48cc-9cdd-d90936b7c07b admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:49.762530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7594b25e-bf53-482f-a720-949df57a6b1c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:49.763138 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7594b25e-bf53-482f-a720-949df57a6b1c service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:49.763414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7594b25e-bf53-482f-a720-949df57a6b1c service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:49.763686 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7594b25e-bf53-482f-a720-949df57a6b1c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:49.773166 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7594b25e-bf53-482f-a720-949df57a6b1c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:49.775184 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7594b25e-bf53-482f-a720-949df57a6b1c service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:49.785585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1236/2479] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:02:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:02:50.201453 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939fd1d2-f71b-41a2-a9e3-8d456464c622 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:50.203067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-939fd1d2-f71b-41a2-a9e3-8d456464c622 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:50.218879 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939fd1d2-f71b-41a2-a9e3-8d456464c622 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:50.223641 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b5b37b-d697-4239-8ea0-63f8fa80ba06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:50.224318 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b5b37b-d697-4239-8ea0-63f8fa80ba06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:50.224654 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b5b37b-d697-4239-8ea0-63f8fa80ba06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:50.224936 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b5b37b-d697-4239-8ea0-63f8fa80ba06 admin admin] PATH_INFO: `/v3/projects/f877d8f0958f4952b607685b1a5cb618` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:50.226079 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b5b37b-d697-4239-8ea0-63f8fa80ba06 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f877d8f0958f4952b607685b1a5cb618)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:50.228897 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b5b37b-d697-4239-8ea0-63f8fa80ba06 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:50.237809 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:50.293261 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-71b5b37b-d697-4239-8ea0-63f8fa80ba06 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f877d8f0958f4952b607685b1a5cb618', 'request_id': 'req-71b5b37b-d697-4239-8ea0-63f8fa80ba06'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:50.299002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-71b5b37b-d697-4239-8ea0-63f8fa80ba06 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f877d8f0958f4952b607685b1a5cb618', 'request_id': 'req-71b5b37b-d697-4239-8ea0-63f8fa80ba06'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:50.306852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-71b5b37b-d697-4239-8ea0-63f8fa80ba06 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f877d8f0958f4952b607685b1a5cb618', 'request_id': 'req-71b5b37b-d697-4239-8ea0-63f8fa80ba06'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:50.318267 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-71b5b37b-d697-4239-8ea0-63f8fa80ba06 admin admin] The token cache is being invalidate because project f877d8f0958f4952b607685b1a5cb618 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:02:50.318593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-71b5b37b-d697-4239-8ea0-63f8fa80ba06 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:50.319272 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-71b5b37b-d697-4239-8ea0-63f8fa80ba06 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:50.321312 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1244/2480] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:02:50 2022] DELETE /identity/v3/projects/f877d8f0958f4952b607685b1a5cb618 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 01:02:50.325811 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7594b25e-bf53-482f-a720-949df57a6b1c service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:50.337951 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7594b25e-bf53-482f-a720-949df57a6b1c service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:50.355840 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:50.398211 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f4201b-e26b-453d-83ee-97d9a3cfc6ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:50.399747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f4201b-e26b-453d-83ee-97d9a3cfc6ff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:50.400035 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f4201b-e26b-453d-83ee-97d9a3cfc6ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:50.400433 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f4201b-e26b-453d-83ee-97d9a3cfc6ff admin admin] PATH_INFO: `/v3/users/58f9818c16f443ffabb285d39a5105be` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:50.401667 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f4201b-e26b-453d-83ee-97d9a3cfc6ff admin admin] RBAC: Authorizing `identity:delete_user(user_id=58f9818c16f443ffabb285d39a5105be)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:50.403638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f4201b-e26b-453d-83ee-97d9a3cfc6ff admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:50.454910 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-f4f4201b-e26b-453d-83ee-97d9a3cfc6ff admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '58f9818c16f443ffabb285d39a5105be'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:50.465455 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-f4f4201b-e26b-453d-83ee-97d9a3cfc6ff admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '58f9818c16f443ffabb285d39a5105be'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:50.467825 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-f4f4201b-e26b-453d-83ee-97d9a3cfc6ff admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '58f9818c16f443ffabb285d39a5105be'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:50.472533 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-f4f4201b-e26b-453d-83ee-97d9a3cfc6ff admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '58f9818c16f443ffabb285d39a5105be'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:50.481708 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-f4f4201b-e26b-453d-83ee-97d9a3cfc6ff admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '58f9818c16f443ffabb285d39a5105be'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:50.501042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1237/2481] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:02:50 2022] DELETE /identity/v3/users/58f9818c16f443ffabb285d39a5105be => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:02:50.770632 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b5b37b-d697-4239-8ea0-63f8fa80ba06 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:50.772062 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-71b5b37b-d697-4239-8ea0-63f8fa80ba06 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:50.785007 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b5b37b-d697-4239-8ea0-63f8fa80ba06 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:50.842489 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcbd3d6-e64b-41fb-8e09-ebf1f9ec82ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:50.843129 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcbd3d6-e64b-41fb-8e09-ebf1f9ec82ff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:50.843405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcbd3d6-e64b-41fb-8e09-ebf1f9ec82ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:50.843672 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcbd3d6-e64b-41fb-8e09-ebf1f9ec82ff admin admin] PATH_INFO: `/v3/projects/8e5dbf82811144e7b8b7cefaf241f4ce` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:50.844761 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcbd3d6-e64b-41fb-8e09-ebf1f9ec82ff admin admin] RBAC: Authorizing `identity:delete_project(project_id=8e5dbf82811144e7b8b7cefaf241f4ce)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:50.846681 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcbd3d6-e64b-41fb-8e09-ebf1f9ec82ff admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:50.892106 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8dcbd3d6-e64b-41fb-8e09-ebf1f9ec82ff admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8e5dbf82811144e7b8b7cefaf241f4ce', 'request_id': 'req-8dcbd3d6-e64b-41fb-8e09-ebf1f9ec82ff'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:50.897494 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8dcbd3d6-e64b-41fb-8e09-ebf1f9ec82ff admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8e5dbf82811144e7b8b7cefaf241f4ce', 'request_id': 'req-8dcbd3d6-e64b-41fb-8e09-ebf1f9ec82ff'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:50.904936 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8dcbd3d6-e64b-41fb-8e09-ebf1f9ec82ff admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8e5dbf82811144e7b8b7cefaf241f4ce', 'request_id': 'req-8dcbd3d6-e64b-41fb-8e09-ebf1f9ec82ff'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:50.915777 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8dcbd3d6-e64b-41fb-8e09-ebf1f9ec82ff admin admin] The token cache is being invalidate because project 8e5dbf82811144e7b8b7cefaf241f4ce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:02:50.916065 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8dcbd3d6-e64b-41fb-8e09-ebf1f9ec82ff admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:50.916692 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8dcbd3d6-e64b-41fb-8e09-ebf1f9ec82ff admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:50.918159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1245/2482] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:02:50 2022] DELETE /identity/v3/projects/8e5dbf82811144e7b8b7cefaf241f4ce => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:02:50.923064 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f12305-638a-4285-ad6c-56087c7923b0 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:50.923354 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f12305-638a-4285-ad6c-56087c7923b0 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:50.923629 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f12305-638a-4285-ad6c-56087c7923b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:50.943570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-c5f12305-638a-4285-ad6c-56087c7923b0 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:02:50.944763 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c5f12305-638a-4285-ad6c-56087c7923b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:51.004021 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1238/2483] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:02:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:02:51.011718 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc2b4a4-14f5-49ba-9b2d-ffc844a2ef4c None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:51.012030 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc2b4a4-14f5-49ba-9b2d-ffc844a2ef4c None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:51.012310 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc2b4a4-14f5-49ba-9b2d-ffc844a2ef4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:51.032066 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-ccc2b4a4-14f5-49ba-9b2d-ffc844a2ef4c None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:02:51.033270 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-ccc2b4a4-14f5-49ba-9b2d-ffc844a2ef4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:51.041878 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1246/2484] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:02:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:02:51.046263 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f12305-638a-4285-ad6c-56087c7923b0 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:51.056958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f12305-638a-4285-ad6c-56087c7923b0 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:51.062264 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20e7efe-6fbb-4a54-bdc5-3636ebf62bef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:51.062853 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20e7efe-6fbb-4a54-bdc5-3636ebf62bef admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:51.063130 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20e7efe-6fbb-4a54-bdc5-3636ebf62bef admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:51.063491 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20e7efe-6fbb-4a54-bdc5-3636ebf62bef admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:51.064049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20e7efe-6fbb-4a54-bdc5-3636ebf62bef admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:02:51.064356 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20e7efe-6fbb-4a54-bdc5-3636ebf62bef admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:51.065964 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20e7efe-6fbb-4a54-bdc5-3636ebf62bef admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:51.078312 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1239/2485] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:02:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 01:02:51.082947 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc2b4a4-14f5-49ba-9b2d-ffc844a2ef4c None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:51.092103 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc2b4a4-14f5-49ba-9b2d-ffc844a2ef4c None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:51.096848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38cc007-2d6e-43bc-9185-8f7c748c3abf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:51.097449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38cc007-2d6e-43bc-9185-8f7c748c3abf admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:51.097729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38cc007-2d6e-43bc-9185-8f7c748c3abf admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:51.098004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38cc007-2d6e-43bc-9185-8f7c748c3abf admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:51.098697 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38cc007-2d6e-43bc-9185-8f7c748c3abf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:51.100631 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38cc007-2d6e-43bc-9185-8f7c748c3abf admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:51.120023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:51.135805 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1247/2486] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:02:51 2022] POST /identity/v3/projects => generated 397 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:02:51.141914 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20e7efe-6fbb-4a54-bdc5-3636ebf62bef admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:51.155591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20e7efe-6fbb-4a54-bdc5-3636ebf62bef admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:51.185506 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:51.220896 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3258839-9a02-4db4-8a03-3c984893c196 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:51.221508 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3258839-9a02-4db4-8a03-3c984893c196 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:51.221788 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3258839-9a02-4db4-8a03-3c984893c196 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:51.222073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3258839-9a02-4db4-8a03-3c984893c196 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:51.222725 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3258839-9a02-4db4-8a03-3c984893c196 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:51.224638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3258839-9a02-4db4-8a03-3c984893c196 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:51.247509 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1240/2487] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:02:51 2022] POST /identity/v3/users => generated 299 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:02:51.251468 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38cc007-2d6e-43bc-9185-8f7c748c3abf admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:51.262780 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38cc007-2d6e-43bc-9185-8f7c748c3abf admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:51.267645 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767a0cf0-3de7-4639-864a-a0a08a86d96f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:51.268244 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767a0cf0-3de7-4639-864a-a0a08a86d96f admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:51.268522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767a0cf0-3de7-4639-864a-a0a08a86d96f admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:51.268793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767a0cf0-3de7-4639-864a-a0a08a86d96f admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:51.269430 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767a0cf0-3de7-4639-864a-a0a08a86d96f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:51.271076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767a0cf0-3de7-4639-864a-a0a08a86d96f admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:51.284915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1248/2488] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:02:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:02:51.286215 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3258839-9a02-4db4-8a03-3c984893c196 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:51.296638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3258839-9a02-4db4-8a03-3c984893c196 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:51.301358 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc547f8d-d057-43ad-874f-ff54c1b554e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:51.301985 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc547f8d-d057-43ad-874f-ff54c1b554e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:51.302243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc547f8d-d057-43ad-874f-ff54c1b554e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:51.302550 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc547f8d-d057-43ad-874f-ff54c1b554e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:51.303368 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc547f8d-d057-43ad-874f-ff54c1b554e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:51.305037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc547f8d-d057-43ad-874f-ff54c1b554e7 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:51.316358 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1241/2489] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:02:51 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:02:51.320302 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767a0cf0-3de7-4639-864a-a0a08a86d96f admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:51.331571 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767a0cf0-3de7-4639-864a-a0a08a86d96f admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:51.337074 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbc2258-6b45-410a-b6c3-46fdebb6a580 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:51.337815 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbc2258-6b45-410a-b6c3-46fdebb6a580 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:51.338099 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbc2258-6b45-410a-b6c3-46fdebb6a580 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:51.338371 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbc2258-6b45-410a-b6c3-46fdebb6a580 admin admin] PATH_INFO: `/v3/projects/74193f3614034419bde02dac15591f38/users/d4c59c6b786a4e1c86c7ff83c0e76d9e/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:51.359494 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efbc2258-6b45-410a-b6c3-46fdebb6a580 admin admin] RBAC: Authorizing `identity:create_grant(project_id=74193f3614034419bde02dac15591f38, user_id=d4c59c6b786a4e1c86c7ff83c0e76d9e, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:51.362280 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efbc2258-6b45-410a-b6c3-46fdebb6a580 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:51.377637 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1249/2490] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:02:51 2022] PUT /identity/v3/projects/74193f3614034419bde02dac15591f38/users/d4c59c6b786a4e1c86c7ff83c0e76d9e/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:02:51.389803 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbdafdd-40d2-48a9-b0d1-84b26ae244e7 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:51.390331 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbdafdd-40d2-48a9-b0d1-84b26ae244e7 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:51.390331 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbdafdd-40d2-48a9-b0d1-84b26ae244e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:51.420176 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-7bbdafdd-40d2-48a9-b0d1-84b26ae244e7 None None] MFA Rules not processed for user `d4c59c6b786a4e1c86c7ff83c0e76d9e`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:02:51.421412 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-7bbdafdd-40d2-48a9-b0d1-84b26ae244e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:51.493045 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1242/2491] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:02:51 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:02:51.497971 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbc2258-6b45-410a-b6c3-46fdebb6a580 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:51.565199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbc2258-6b45-410a-b6c3-46fdebb6a580 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:51.569800 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7608751-68e1-42b4-b9f6-bdb88691a843 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:51.571081 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7608751-68e1-42b4-b9f6-bdb88691a843 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:51.571360 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7608751-68e1-42b4-b9f6-bdb88691a843 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:51.571629 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7608751-68e1-42b4-b9f6-bdb88691a843 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:51.572251 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7608751-68e1-42b4-b9f6-bdb88691a843 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:51.574138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7608751-68e1-42b4-b9f6-bdb88691a843 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:51.600503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1250/2492] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:02:51 2022] POST /identity/v3/projects => generated 401 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:02:51.604583 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbdafdd-40d2-48a9-b0d1-84b26ae244e7 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:51.615217 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbdafdd-40d2-48a9-b0d1-84b26ae244e7 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:51.670038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061387be-7d8a-4b1a-b025-70f3b6e0a5ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:51.670722 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061387be-7d8a-4b1a-b025-70f3b6e0a5ce admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:51.670948 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061387be-7d8a-4b1a-b025-70f3b6e0a5ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:51.671219 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061387be-7d8a-4b1a-b025-70f3b6e0a5ce admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:51.671838 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061387be-7d8a-4b1a-b025-70f3b6e0a5ce admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:51.673739 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061387be-7d8a-4b1a-b025-70f3b6e0a5ce admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:51.694978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1243/2493] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:02:51 2022] POST /identity/v3/users => generated 307 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:02:51.699697 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7608751-68e1-42b4-b9f6-bdb88691a843 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:51.710121 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7608751-68e1-42b4-b9f6-bdb88691a843 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:51.714879 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ec53f1-50c4-4923-b0d3-691af9cedbd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:51.715475 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ec53f1-50c4-4923-b0d3-691af9cedbd4 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:51.715754 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ec53f1-50c4-4923-b0d3-691af9cedbd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:51.716026 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ec53f1-50c4-4923-b0d3-691af9cedbd4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:51.716665 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ec53f1-50c4-4923-b0d3-691af9cedbd4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:51.718257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ec53f1-50c4-4923-b0d3-691af9cedbd4 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:51.729126 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1251/2494] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:02:51 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 01:02:51.733191 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061387be-7d8a-4b1a-b025-70f3b6e0a5ce admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:51.744692 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061387be-7d8a-4b1a-b025-70f3b6e0a5ce admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:51.750890 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa91e794-fa11-4c78-bc84-cf454a15dc39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:51.751593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa91e794-fa11-4c78-bc84-cf454a15dc39 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:51.752311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa91e794-fa11-4c78-bc84-cf454a15dc39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:51.752598 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa91e794-fa11-4c78-bc84-cf454a15dc39 admin admin] PATH_INFO: `/v3/projects/ddec05ac235b4480b7a288efaaab0eef/users/339a6656e1fb4e27af0683eff2b7d772/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:51.769255 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa91e794-fa11-4c78-bc84-cf454a15dc39 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ddec05ac235b4480b7a288efaaab0eef, user_id=339a6656e1fb4e27af0683eff2b7d772, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:51.772183 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa91e794-fa11-4c78-bc84-cf454a15dc39 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:51.786869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1244/2495] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:02:51 2022] PUT /identity/v3/projects/ddec05ac235b4480b7a288efaaab0eef/users/339a6656e1fb4e27af0683eff2b7d772/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:02:51.791465 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ec53f1-50c4-4923-b0d3-691af9cedbd4 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:51.802731 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ec53f1-50c4-4923-b0d3-691af9cedbd4 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:51.862663 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c722106-516a-41d2-8ccd-d94df832a89a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:51.863462 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c722106-516a-41d2-8ccd-d94df832a89a admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:51.863746 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c722106-516a-41d2-8ccd-d94df832a89a admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:51.864013 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c722106-516a-41d2-8ccd-d94df832a89a admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:51.864772 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c722106-516a-41d2-8ccd-d94df832a89a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:51.866376 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c722106-516a-41d2-8ccd-d94df832a89a admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:51.878014 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1252/2496] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:02:51 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:02:51.882136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa91e794-fa11-4c78-bc84-cf454a15dc39 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:51.887674 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:51.895779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa91e794-fa11-4c78-bc84-cf454a15dc39 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:51.901329 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d9afc2-4d11-44ae-8112-52056e628da3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:51.901982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d9afc2-4d11-44ae-8112-52056e628da3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:51.902262 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d9afc2-4d11-44ae-8112-52056e628da3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:51.902561 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d9afc2-4d11-44ae-8112-52056e628da3 admin admin] PATH_INFO: `/v3/projects/ddec05ac235b4480b7a288efaaab0eef/users/339a6656e1fb4e27af0683eff2b7d772/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:51.904459 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d9afc2-4d11-44ae-8112-52056e628da3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ddec05ac235b4480b7a288efaaab0eef, user_id=339a6656e1fb4e27af0683eff2b7d772, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:51.907090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d9afc2-4d11-44ae-8112-52056e628da3 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:51.918377 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1245/2497] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:02:51 2022] PUT /identity/v3/projects/ddec05ac235b4480b7a288efaaab0eef/users/339a6656e1fb4e27af0683eff2b7d772/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:02:51.928844 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0090590e-6dd2-471a-9590-283bdfab7180 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:51.929137 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0090590e-6dd2-471a-9590-283bdfab7180 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:51.929411 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0090590e-6dd2-471a-9590-283bdfab7180 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:51.958151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-0090590e-6dd2-471a-9590-283bdfab7180 None None] MFA Rules not processed for user `339a6656e1fb4e27af0683eff2b7d772`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:02:51.959416 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-0090590e-6dd2-471a-9590-283bdfab7180 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:51.979508 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:52.038775 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1253/2498] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:02:51 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:02:52.049912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d9afc2-4d11-44ae-8112-52056e628da3 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:52.051751 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-99d9afc2-4d11-44ae-8112-52056e628da3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:52.112375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d9afc2-4d11-44ae-8112-52056e628da3 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:52.115982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0809f694-e6fd-4473-a81c-c2689f190395 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:52.116583 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0809f694-e6fd-4473-a81c-c2689f190395 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:52.116891 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0809f694-e6fd-4473-a81c-c2689f190395 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:52.117126 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0809f694-e6fd-4473-a81c-c2689f190395 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:52.125921 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0809f694-e6fd-4473-a81c-c2689f190395 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:52.127374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0809f694-e6fd-4473-a81c-c2689f190395 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:52.136496 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1246/2499] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:02:52 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:02:52.217085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0090590e-6dd2-471a-9590-283bdfab7180 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:52.227398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0090590e-6dd2-471a-9590-283bdfab7180 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:52.231535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57df16b-c7de-4e0e-8f18-10f6cbeaccec service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:52.232124 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57df16b-c7de-4e0e-8f18-10f6cbeaccec service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:52.232384 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57df16b-c7de-4e0e-8f18-10f6cbeaccec service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:52.232653 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57df16b-c7de-4e0e-8f18-10f6cbeaccec service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:52.305808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57df16b-c7de-4e0e-8f18-10f6cbeaccec service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:52.307863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57df16b-c7de-4e0e-8f18-10f6cbeaccec service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:52.318351 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1254/2500] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:02:52 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:02:56.018927 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086206c4-fe75-4eb5-8ea4-67e7823b4712 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:56.019415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086206c4-fe75-4eb5-8ea4-67e7823b4712 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:56.019549 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086206c4-fe75-4eb5-8ea4-67e7823b4712 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:56.038839 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-086206c4-fe75-4eb5-8ea4-67e7823b4712 None None] MFA Rules not processed for user `3e31dce69c344c9aa4a55f31b9c735df`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:02:56.040150 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-086206c4-fe75-4eb5-8ea4-67e7823b4712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:56.098608 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1247/2501] 10.222.0.42 () {58 vars in 967 bytes} [Sun Sep 18 01:02:56 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:02:56.148939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57df16b-c7de-4e0e-8f18-10f6cbeaccec service cinder] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:56.150403 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-f57df16b-c7de-4e0e-8f18-10f6cbeaccec service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:56.220306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57df16b-c7de-4e0e-8f18-10f6cbeaccec service cinder] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:56.225261 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68cf560-8bb6-417d-badd-05916d54adb4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6230ed7ce0eb43e9b2b8530a28eb6299', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:56.225869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68cf560-8bb6-417d-badd-05916d54adb4 service swift] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:56.226148 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68cf560-8bb6-417d-badd-05916d54adb4 service swift] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:56.226419 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68cf560-8bb6-417d-badd-05916d54adb4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:56.235372 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68cf560-8bb6-417d-badd-05916d54adb4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:56.237040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68cf560-8bb6-417d-badd-05916d54adb4 service swift] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:56.246261 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1255/2502] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 18 01:02:56 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 18 01:02:56.578126 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086206c4-fe75-4eb5-8ea4-67e7823b4712 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:56.651952 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086206c4-fe75-4eb5-8ea4-67e7823b4712 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:56.657499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3ae687-f1ea-44b7-86c6-eaf7efb959e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:56.658368 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3ae687-f1ea-44b7-86c6-eaf7efb959e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:56.658711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3ae687-f1ea-44b7-86c6-eaf7efb959e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:56.659021 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3ae687-f1ea-44b7-86c6-eaf7efb959e7 admin admin] PATH_INFO: `/v3/users/d4c59c6b786a4e1c86c7ff83c0e76d9e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:56.660675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3ae687-f1ea-44b7-86c6-eaf7efb959e7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d4c59c6b786a4e1c86c7ff83c0e76d9e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:56.662806 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3ae687-f1ea-44b7-86c6-eaf7efb959e7 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:56.724769 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=94,keystone:INSERT=1,keystone:DELETE=5 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:56.741244 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6a3ae687-f1ea-44b7-86c6-eaf7efb959e7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd4c59c6b786a4e1c86c7ff83c0e76d9e'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:56.752147 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6a3ae687-f1ea-44b7-86c6-eaf7efb959e7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd4c59c6b786a4e1c86c7ff83c0e76d9e'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:56.752735 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6a3ae687-f1ea-44b7-86c6-eaf7efb959e7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd4c59c6b786a4e1c86c7ff83c0e76d9e'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:56.753481 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6a3ae687-f1ea-44b7-86c6-eaf7efb959e7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd4c59c6b786a4e1c86c7ff83c0e76d9e'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:56.763351 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6a3ae687-f1ea-44b7-86c6-eaf7efb959e7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd4c59c6b786a4e1c86c7ff83c0e76d9e'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:56.790492 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1248/2503] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:02:56 2022] DELETE /identity/v3/users/d4c59c6b786a4e1c86c7ff83c0e76d9e => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:02:56.794238 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82d1c6e-77e5-4254-8fd0-efa1ad11d47d None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:56.795308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82d1c6e-77e5-4254-8fd0-efa1ad11d47d None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:56.795696 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82d1c6e-77e5-4254-8fd0-efa1ad11d47d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:56.816599 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-a82d1c6e-77e5-4254-8fd0-efa1ad11d47d None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:02:56.818403 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a82d1c6e-77e5-4254-8fd0-efa1ad11d47d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:56.883817 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:56.910644 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1256/2504] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:02:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:02:56.921976 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3ae687-f1ea-44b7-86c6-eaf7efb959e7 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:56.924243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-6a3ae687-f1ea-44b7-86c6-eaf7efb959e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:57.044377 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3ae687-f1ea-44b7-86c6-eaf7efb959e7 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:57.055095 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4878b245-97ac-46c3-bf48-7fd17d1850e5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:57.057334 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4878b245-97ac-46c3-bf48-7fd17d1850e5 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:57.058077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4878b245-97ac-46c3-bf48-7fd17d1850e5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:57.058828 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4878b245-97ac-46c3-bf48-7fd17d1850e5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:57.080903 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4878b245-97ac-46c3-bf48-7fd17d1850e5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:57.084685 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4878b245-97ac-46c3-bf48-7fd17d1850e5 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:57.102034 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1249/2505] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:02:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 01:02:57.526180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82d1c6e-77e5-4254-8fd0-efa1ad11d47d None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:57.527959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a82d1c6e-77e5-4254-8fd0-efa1ad11d47d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:57.549344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82d1c6e-77e5-4254-8fd0-efa1ad11d47d None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:57.556422 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1506019b-ff5f-4d6a-882b-e3c3146fcdcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:57.557084 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1506019b-ff5f-4d6a-882b-e3c3146fcdcd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:57.557396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1506019b-ff5f-4d6a-882b-e3c3146fcdcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:57.557618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1506019b-ff5f-4d6a-882b-e3c3146fcdcd admin admin] PATH_INFO: `/v3/projects/74193f3614034419bde02dac15591f38` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:57.558812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1506019b-ff5f-4d6a-882b-e3c3146fcdcd admin admin] RBAC: Authorizing `identity:delete_project(project_id=74193f3614034419bde02dac15591f38)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:57.560811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1506019b-ff5f-4d6a-882b-e3c3146fcdcd admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:57.622232 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-1506019b-ff5f-4d6a-882b-e3c3146fcdcd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '74193f3614034419bde02dac15591f38', 'request_id': 'req-1506019b-ff5f-4d6a-882b-e3c3146fcdcd'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:57.628307 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-1506019b-ff5f-4d6a-882b-e3c3146fcdcd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '74193f3614034419bde02dac15591f38', 'request_id': 'req-1506019b-ff5f-4d6a-882b-e3c3146fcdcd'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:57.637144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-1506019b-ff5f-4d6a-882b-e3c3146fcdcd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '74193f3614034419bde02dac15591f38', 'request_id': 'req-1506019b-ff5f-4d6a-882b-e3c3146fcdcd'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:57.649180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-1506019b-ff5f-4d6a-882b-e3c3146fcdcd admin admin] The token cache is being invalidate because project 74193f3614034419bde02dac15591f38 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:02:57.649456 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-1506019b-ff5f-4d6a-882b-e3c3146fcdcd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:57.650552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-1506019b-ff5f-4d6a-882b-e3c3146fcdcd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:57.651796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1257/2506] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:02:57 2022] DELETE /identity/v3/projects/74193f3614034419bde02dac15591f38 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:02:57.655711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4878b245-97ac-46c3-bf48-7fd17d1850e5 service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:57.665841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4878b245-97ac-46c3-bf48-7fd17d1850e5 service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:57.737811 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a9c063-39c3-4408-8f2f-fe505a6a3fd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:57.738444 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a9c063-39c3-4408-8f2f-fe505a6a3fd0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:57.738757 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a9c063-39c3-4408-8f2f-fe505a6a3fd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:57.739029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a9c063-39c3-4408-8f2f-fe505a6a3fd0 admin admin] PATH_INFO: `/v3/users/339a6656e1fb4e27af0683eff2b7d772` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:57.740169 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a9c063-39c3-4408-8f2f-fe505a6a3fd0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=339a6656e1fb4e27af0683eff2b7d772)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:57.742121 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a9c063-39c3-4408-8f2f-fe505a6a3fd0 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:57.804251 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c2a9c063-39c3-4408-8f2f-fe505a6a3fd0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '339a6656e1fb4e27af0683eff2b7d772'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:57.819942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c2a9c063-39c3-4408-8f2f-fe505a6a3fd0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '339a6656e1fb4e27af0683eff2b7d772'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:57.820652 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c2a9c063-39c3-4408-8f2f-fe505a6a3fd0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '339a6656e1fb4e27af0683eff2b7d772'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:57.821263 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c2a9c063-39c3-4408-8f2f-fe505a6a3fd0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '339a6656e1fb4e27af0683eff2b7d772'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:57.827739 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:DELETE=13,keystone:SELECT=85,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:57.831228 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c2a9c063-39c3-4408-8f2f-fe505a6a3fd0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '339a6656e1fb4e27af0683eff2b7d772'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:57.861580 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1250/2507] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:02:57 2022] DELETE /identity/v3/users/339a6656e1fb4e27af0683eff2b7d772 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:02:58.375737 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1506019b-ff5f-4d6a-882b-e3c3146fcdcd admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:58.377178 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-1506019b-ff5f-4d6a-882b-e3c3146fcdcd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:58.395114 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1506019b-ff5f-4d6a-882b-e3c3146fcdcd admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:58.467728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37f850f-d3cd-4966-9c31-53b97565b23a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:58.468433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37f850f-d3cd-4966-9c31-53b97565b23a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:58.469853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37f850f-d3cd-4966-9c31-53b97565b23a admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:58.469853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37f850f-d3cd-4966-9c31-53b97565b23a admin admin] PATH_INFO: `/v3/projects/ddec05ac235b4480b7a288efaaab0eef` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:58.470334 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37f850f-d3cd-4966-9c31-53b97565b23a admin admin] RBAC: Authorizing `identity:delete_project(project_id=ddec05ac235b4480b7a288efaaab0eef)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:58.472375 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37f850f-d3cd-4966-9c31-53b97565b23a admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:58.536147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-e37f850f-d3cd-4966-9c31-53b97565b23a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ddec05ac235b4480b7a288efaaab0eef', 'request_id': 'req-e37f850f-d3cd-4966-9c31-53b97565b23a'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:58.544094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-e37f850f-d3cd-4966-9c31-53b97565b23a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ddec05ac235b4480b7a288efaaab0eef', 'request_id': 'req-e37f850f-d3cd-4966-9c31-53b97565b23a'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:58.553959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-e37f850f-d3cd-4966-9c31-53b97565b23a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ddec05ac235b4480b7a288efaaab0eef', 'request_id': 'req-e37f850f-d3cd-4966-9c31-53b97565b23a'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:58.563525 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:58.569102 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-e37f850f-d3cd-4966-9c31-53b97565b23a admin admin] The token cache is being invalidate because project ddec05ac235b4480b7a288efaaab0eef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:02:58.569982 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-e37f850f-d3cd-4966-9c31-53b97565b23a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:58.571653 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-e37f850f-d3cd-4966-9c31-53b97565b23a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:02:58.574901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1258/2508] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:02:58 2022] DELETE /identity/v3/projects/ddec05ac235b4480b7a288efaaab0eef => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:02:58.580875 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21c1174-281b-4b14-abea-a9aceee77e96 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:58.581633 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21c1174-281b-4b14-abea-a9aceee77e96 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:58.582201 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21c1174-281b-4b14-abea-a9aceee77e96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:58.604397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-c21c1174-281b-4b14-abea-a9aceee77e96 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:02:58.605746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c21c1174-281b-4b14-abea-a9aceee77e96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:58.686752 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1251/2509] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:02:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:02:58.688572 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9525f82b-8034-45bd-ade6-4b8baa47f954 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:58.688964 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9525f82b-8034-45bd-ade6-4b8baa47f954 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:58.689836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9525f82b-8034-45bd-ade6-4b8baa47f954 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:58.719277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-9525f82b-8034-45bd-ade6-4b8baa47f954 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:02:58.722197 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-9525f82b-8034-45bd-ade6-4b8baa47f954 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:58.736371 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1259/2510] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:02:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:02:58.738019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21c1174-281b-4b14-abea-a9aceee77e96 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:58.747799 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21c1174-281b-4b14-abea-a9aceee77e96 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:58.752465 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216fe625-43a3-45cc-bb6f-87d77b49b39a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:58.753073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216fe625-43a3-45cc-bb6f-87d77b49b39a admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:58.753348 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216fe625-43a3-45cc-bb6f-87d77b49b39a admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:58.753616 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216fe625-43a3-45cc-bb6f-87d77b49b39a admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:58.754266 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216fe625-43a3-45cc-bb6f-87d77b49b39a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:02:58.755086 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216fe625-43a3-45cc-bb6f-87d77b49b39a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:58.756225 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216fe625-43a3-45cc-bb6f-87d77b49b39a admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:58.769122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1252/2511] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:02:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 01:02:58.773453 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9525f82b-8034-45bd-ade6-4b8baa47f954 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:58.783856 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9525f82b-8034-45bd-ade6-4b8baa47f954 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:58.789354 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592fc807-a9a5-4212-a15c-2e36432b808c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:58.789974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592fc807-a9a5-4212-a15c-2e36432b808c admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:58.790253 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592fc807-a9a5-4212-a15c-2e36432b808c admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:58.790983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592fc807-a9a5-4212-a15c-2e36432b808c admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:58.791371 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592fc807-a9a5-4212-a15c-2e36432b808c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:58.795768 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592fc807-a9a5-4212-a15c-2e36432b808c admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:58.829037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1260/2512] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:02:58 2022] POST /identity/v3/projects => generated 367 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:02:58.833524 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216fe625-43a3-45cc-bb6f-87d77b49b39a admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:58.843960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216fe625-43a3-45cc-bb6f-87d77b49b39a admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:58.921829 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca2e8ae-716c-4c10-ae82-5b434176cd4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:58.922450 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca2e8ae-716c-4c10-ae82-5b434176cd4f admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:58.922820 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca2e8ae-716c-4c10-ae82-5b434176cd4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:58.923057 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca2e8ae-716c-4c10-ae82-5b434176cd4f admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:58.923764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca2e8ae-716c-4c10-ae82-5b434176cd4f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:58.927364 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca2e8ae-716c-4c10-ae82-5b434176cd4f admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:58.965910 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1253/2513] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:02:58 2022] POST /identity/v3/users => generated 284 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 01:02:58.967688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592fc807-a9a5-4212-a15c-2e36432b808c admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:58.980862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592fc807-a9a5-4212-a15c-2e36432b808c admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:58.986354 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d343175a-37ef-4441-b4f2-81d7bf5f9f92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:58.988067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d343175a-37ef-4441-b4f2-81d7bf5f9f92 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:58.988443 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d343175a-37ef-4441-b4f2-81d7bf5f9f92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:58.988569 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d343175a-37ef-4441-b4f2-81d7bf5f9f92 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:58.989205 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d343175a-37ef-4441-b4f2-81d7bf5f9f92 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:58.991124 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d343175a-37ef-4441-b4f2-81d7bf5f9f92 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:59.005433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1261/2514] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:02:58 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:02:59.009958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca2e8ae-716c-4c10-ae82-5b434176cd4f admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:59.020517 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca2e8ae-716c-4c10-ae82-5b434176cd4f admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:59.025881 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a84e2c-508d-4d5c-be15-1d0a6a078b06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:59.026533 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a84e2c-508d-4d5c-be15-1d0a6a078b06 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:59.026816 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a84e2c-508d-4d5c-be15-1d0a6a078b06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:59.027092 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a84e2c-508d-4d5c-be15-1d0a6a078b06 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:59.027811 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a84e2c-508d-4d5c-be15-1d0a6a078b06 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:59.031063 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a84e2c-508d-4d5c-be15-1d0a6a078b06 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:59.042624 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1254/2515] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:02:59 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:02:59.046879 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d343175a-37ef-4441-b4f2-81d7bf5f9f92 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:59.060160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d343175a-37ef-4441-b4f2-81d7bf5f9f92 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:59.067154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8afe323-7fa6-466c-86f0-df2f55c339b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:59.067857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8afe323-7fa6-466c-86f0-df2f55c339b9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:59.068175 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8afe323-7fa6-466c-86f0-df2f55c339b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:59.068514 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8afe323-7fa6-466c-86f0-df2f55c339b9 admin admin] PATH_INFO: `/v3/projects/de7c0f943cc14eb38b908863f391241a/users/2f54379006454d6da18c0e6d1682b9c6/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:59.086854 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8afe323-7fa6-466c-86f0-df2f55c339b9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=de7c0f943cc14eb38b908863f391241a, user_id=2f54379006454d6da18c0e6d1682b9c6, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:59.089703 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8afe323-7fa6-466c-86f0-df2f55c339b9 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:59.106156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1262/2516] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:02:59 2022] PUT /identity/v3/projects/de7c0f943cc14eb38b908863f391241a/users/2f54379006454d6da18c0e6d1682b9c6/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:02:59.117117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b88e90-9f7b-43c6-954f-084e003c94e3 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:59.117904 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b88e90-9f7b-43c6-954f-084e003c94e3 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:59.118174 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b88e90-9f7b-43c6-954f-084e003c94e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:59.147106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-f3b88e90-9f7b-43c6-954f-084e003c94e3 None None] MFA Rules not processed for user `2f54379006454d6da18c0e6d1682b9c6`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:02:59.148636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-f3b88e90-9f7b-43c6-954f-084e003c94e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:59.186220 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:59.225445 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1255/2517] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:02:59 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:02:59.231169 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8afe323-7fa6-466c-86f0-df2f55c339b9 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:59.322979 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8afe323-7fa6-466c-86f0-df2f55c339b9 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:59.333155 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48bdaac0-6fe4-42e9-af1d-d573984e2839 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:59.334167 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bdaac0-6fe4-42e9-af1d-d573984e2839 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:59.334675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bdaac0-6fe4-42e9-af1d-d573984e2839 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:59.335247 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bdaac0-6fe4-42e9-af1d-d573984e2839 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:59.336067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48bdaac0-6fe4-42e9-af1d-d573984e2839 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:59.338998 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48bdaac0-6fe4-42e9-af1d-d573984e2839 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:59.382516 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1263/2518] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:02:59 2022] POST /identity/v3/projects => generated 367 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:02:59.384420 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b88e90-9f7b-43c6-954f-084e003c94e3 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:59.400327 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b88e90-9f7b-43c6-954f-084e003c94e3 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:59.469992 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e511286f-199c-43d8-823c-ecaf82d43af7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:59.470730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e511286f-199c-43d8-823c-ecaf82d43af7 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:59.471079 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e511286f-199c-43d8-823c-ecaf82d43af7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:59.471421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e511286f-199c-43d8-823c-ecaf82d43af7 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:59.472131 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e511286f-199c-43d8-823c-ecaf82d43af7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:59.474123 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e511286f-199c-43d8-823c-ecaf82d43af7 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:59.499360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1256/2519] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:02:59 2022] POST /identity/v3/users => generated 290 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:02:59.500638 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48bdaac0-6fe4-42e9-af1d-d573984e2839 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:59.512199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48bdaac0-6fe4-42e9-af1d-d573984e2839 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:59.517177 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f104a5e5-5cf2-46ca-96f0-a7d04566f7a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:59.517906 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f104a5e5-5cf2-46ca-96f0-a7d04566f7a6 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:59.518186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f104a5e5-5cf2-46ca-96f0-a7d04566f7a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:59.518485 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f104a5e5-5cf2-46ca-96f0-a7d04566f7a6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:59.519146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f104a5e5-5cf2-46ca-96f0-a7d04566f7a6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:59.520856 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f104a5e5-5cf2-46ca-96f0-a7d04566f7a6 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:59.533162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1264/2520] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:02:59 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 01:02:59.538263 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e511286f-199c-43d8-823c-ecaf82d43af7 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:59.550120 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e511286f-199c-43d8-823c-ecaf82d43af7 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:59.555958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3695fd-f38d-4c67-b640-7c6e58740952 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:59.556637 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3695fd-f38d-4c67-b640-7c6e58740952 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:59.556916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3695fd-f38d-4c67-b640-7c6e58740952 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:59.557217 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3695fd-f38d-4c67-b640-7c6e58740952 admin admin] PATH_INFO: `/v3/projects/5d944a07614f4a94a84194803db359bd/users/e879c9752f15473cbbfabee5c21decc9/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:59.584361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3695fd-f38d-4c67-b640-7c6e58740952 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d944a07614f4a94a84194803db359bd, user_id=e879c9752f15473cbbfabee5c21decc9, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:59.587650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3695fd-f38d-4c67-b640-7c6e58740952 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:59.605771 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1257/2521] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:02:59 2022] PUT /identity/v3/projects/5d944a07614f4a94a84194803db359bd/users/e879c9752f15473cbbfabee5c21decc9/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:02:59.608987 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f104a5e5-5cf2-46ca-96f0-a7d04566f7a6 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:59.623340 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f104a5e5-5cf2-46ca-96f0-a7d04566f7a6 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:59.692587 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c9b804-5e76-4e80-83e7-c4c74ac0a081 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:59.693544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c9b804-5e76-4e80-83e7-c4c74ac0a081 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:59.693992 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c9b804-5e76-4e80-83e7-c4c74ac0a081 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:59.694416 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c9b804-5e76-4e80-83e7-c4c74ac0a081 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:59.695236 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c9b804-5e76-4e80-83e7-c4c74ac0a081 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:59.697250 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c9b804-5e76-4e80-83e7-c4c74ac0a081 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:59.710617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1265/2522] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:02:59 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:02:59.712750 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3695fd-f38d-4c67-b640-7c6e58740952 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:59.722966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3695fd-f38d-4c67-b640-7c6e58740952 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:02:59.728179 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee8b0e6-668c-4d57-85cf-d20a64abb031 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:02:59.728855 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee8b0e6-668c-4d57-85cf-d20a64abb031 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:59.729133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee8b0e6-668c-4d57-85cf-d20a64abb031 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:59.729406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee8b0e6-668c-4d57-85cf-d20a64abb031 admin admin] PATH_INFO: `/v3/projects/5d944a07614f4a94a84194803db359bd/users/e879c9752f15473cbbfabee5c21decc9/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:59.731375 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee8b0e6-668c-4d57-85cf-d20a64abb031 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d944a07614f4a94a84194803db359bd, user_id=e879c9752f15473cbbfabee5c21decc9, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:02:59.733993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee8b0e6-668c-4d57-85cf-d20a64abb031 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:02:59.748266 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1258/2523] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:02:59 2022] PUT /identity/v3/projects/5d944a07614f4a94a84194803db359bd/users/e879c9752f15473cbbfabee5c21decc9/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:02:59.758758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78063a2-5108-42ba-b752-61fcf46cbe91 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:02:59.759123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78063a2-5108-42ba-b752-61fcf46cbe91 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:02:59.759551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78063a2-5108-42ba-b752-61fcf46cbe91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:02:59.768829 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:02:59.803750 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-d78063a2-5108-42ba-b752-61fcf46cbe91 None None] MFA Rules not processed for user `e879c9752f15473cbbfabee5c21decc9`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:02:59.805062 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-d78063a2-5108-42ba-b752-61fcf46cbe91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:02:59.895602 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1266/2524] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:02:59 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:02:59.914122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee8b0e6-668c-4d57-85cf-d20a64abb031 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:02:59.918446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-9ee8b0e6-668c-4d57-85cf-d20a64abb031 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:00.000527 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee8b0e6-668c-4d57-85cf-d20a64abb031 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:00.005326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95943ed-4bd6-46b1-8caa-22486f529080 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:00.006060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95943ed-4bd6-46b1-8caa-22486f529080 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:00.006439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95943ed-4bd6-46b1-8caa-22486f529080 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:00.007604 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95943ed-4bd6-46b1-8caa-22486f529080 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:00.020037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95943ed-4bd6-46b1-8caa-22486f529080 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:00.021646 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95943ed-4bd6-46b1-8caa-22486f529080 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:00.036260 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1259/2525] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:02:59 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:03:00.153106 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78063a2-5108-42ba-b752-61fcf46cbe91 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:00.164227 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78063a2-5108-42ba-b752-61fcf46cbe91 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:00.171458 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230e3553-a46f-4448-a67c-3d8237582dad service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:00.171660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230e3553-a46f-4448-a67c-3d8237582dad service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:00.171964 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230e3553-a46f-4448-a67c-3d8237582dad service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:00.172304 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230e3553-a46f-4448-a67c-3d8237582dad service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:00.271801 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230e3553-a46f-4448-a67c-3d8237582dad service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:00.274415 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230e3553-a46f-4448-a67c-3d8237582dad service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:00.284466 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1267/2526] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:03:00 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:03:03.371244 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd49631c-8b4c-4526-8fc1-9455b10763a6 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:03.371712 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd49631c-8b4c-4526-8fc1-9455b10763a6 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:03.371869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd49631c-8b4c-4526-8fc1-9455b10763a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:03.407805 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-dd49631c-8b4c-4526-8fc1-9455b10763a6 None None] MFA Rules not processed for user `3e31dce69c344c9aa4a55f31b9c735df`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:03.409024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-dd49631c-8b4c-4526-8fc1-9455b10763a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:03.454608 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:03.500230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1260/2527] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 18 01:03:03 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:03:03.519055 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230e3553-a46f-4448-a67c-3d8237582dad service cinder] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:03.522816 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-230e3553-a46f-4448-a67c-3d8237582dad service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:03.525571 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd49631c-8b4c-4526-8fc1-9455b10763a6 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:03.528114 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-dd49631c-8b4c-4526-8fc1-9455b10763a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:03.641786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd49631c-8b4c-4526-8fc1-9455b10763a6 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:03.654244 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d16e211-47a0-4b94-81e4-1a7b1e12b397 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:03.654917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d16e211-47a0-4b94-81e4-1a7b1e12b397 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:03.655145 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d16e211-47a0-4b94-81e4-1a7b1e12b397 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:03.655415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d16e211-47a0-4b94-81e4-1a7b1e12b397 admin admin] PATH_INFO: `/v3/users/20a50c6db67f4d818a6fbeaafce299d9` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:03.657024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d16e211-47a0-4b94-81e4-1a7b1e12b397 admin admin] RBAC: Authorizing `identity:delete_user(user_id=20a50c6db67f4d818a6fbeaafce299d9)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:03.659278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d16e211-47a0-4b94-81e4-1a7b1e12b397 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:03.669387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230e3553-a46f-4448-a67c-3d8237582dad service cinder] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:03.677385 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7caa7b-ec3a-44eb-9e87-6ad0fd4de02b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6230ed7ce0eb43e9b2b8530a28eb6299', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:03.686200 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7caa7b-ec3a-44eb-9e87-6ad0fd4de02b service swift] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:03.686200 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7caa7b-ec3a-44eb-9e87-6ad0fd4de02b service swift] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:03.686200 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7caa7b-ec3a-44eb-9e87-6ad0fd4de02b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:03.701529 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7caa7b-ec3a-44eb-9e87-6ad0fd4de02b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:03.703210 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7caa7b-ec3a-44eb-9e87-6ad0fd4de02b service swift] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:03.715785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1268/2528] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 18 01:03:03 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 18 01:03:03.726408 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8d16e211-47a0-4b94-81e4-1a7b1e12b397 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '20a50c6db67f4d818a6fbeaafce299d9'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:03.734652 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8d16e211-47a0-4b94-81e4-1a7b1e12b397 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '20a50c6db67f4d818a6fbeaafce299d9'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:03.735374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8d16e211-47a0-4b94-81e4-1a7b1e12b397 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '20a50c6db67f4d818a6fbeaafce299d9'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:03.742736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8d16e211-47a0-4b94-81e4-1a7b1e12b397 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '20a50c6db67f4d818a6fbeaafce299d9'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:03.752997 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8d16e211-47a0-4b94-81e4-1a7b1e12b397 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '20a50c6db67f4d818a6fbeaafce299d9'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:03.785132 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1261/2529] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:03:03 2022] DELETE /identity/v3/users/20a50c6db67f4d818a6fbeaafce299d9 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:03:04.084611 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7caa7b-ec3a-44eb-9e87-6ad0fd4de02b service swift] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:04.098987 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7caa7b-ec3a-44eb-9e87-6ad0fd4de02b service swift] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:04.127519 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=100 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:04.205107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a82d5d6-9b96-4817-beac-6e8238de4163 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:04.221067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a82d5d6-9b96-4817-beac-6e8238de4163 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:04.221910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a82d5d6-9b96-4817-beac-6e8238de4163 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:04.222208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a82d5d6-9b96-4817-beac-6e8238de4163 admin admin] PATH_INFO: `/v3/projects/c9f737f6350f4d1f993708b1f90a862e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:04.226118 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a82d5d6-9b96-4817-beac-6e8238de4163 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c9f737f6350f4d1f993708b1f90a862e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:04.233711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a82d5d6-9b96-4817-beac-6e8238de4163 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:04.384141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2a82d5d6-9b96-4817-beac-6e8238de4163 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c9f737f6350f4d1f993708b1f90a862e', 'request_id': 'req-2a82d5d6-9b96-4817-beac-6e8238de4163'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:04.393628 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2a82d5d6-9b96-4817-beac-6e8238de4163 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c9f737f6350f4d1f993708b1f90a862e', 'request_id': 'req-2a82d5d6-9b96-4817-beac-6e8238de4163'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:04.406220 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2a82d5d6-9b96-4817-beac-6e8238de4163 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c9f737f6350f4d1f993708b1f90a862e', 'request_id': 'req-2a82d5d6-9b96-4817-beac-6e8238de4163'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:04.425380 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2a82d5d6-9b96-4817-beac-6e8238de4163 admin admin] The token cache is being invalidate because project c9f737f6350f4d1f993708b1f90a862e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:03:04.425721 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2a82d5d6-9b96-4817-beac-6e8238de4163 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:04.426520 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2a82d5d6-9b96-4817-beac-6e8238de4163 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:04.433888 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1269/2530] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:03:04 2022] DELETE /identity/v3/projects/c9f737f6350f4d1f993708b1f90a862e => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:03:04.434827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d16e211-47a0-4b94-81e4-1a7b1e12b397 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:04.437153 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8d16e211-47a0-4b94-81e4-1a7b1e12b397 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:04.461508 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d16e211-47a0-4b94-81e4-1a7b1e12b397 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:04.580251 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:04.598885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4453001c-f1d1-4b10-8fd9-7dccec980074 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:04.604236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4453001c-f1d1-4b10-8fd9-7dccec980074 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:04.605008 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4453001c-f1d1-4b10-8fd9-7dccec980074 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:04.605669 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4453001c-f1d1-4b10-8fd9-7dccec980074 admin admin] PATH_INFO: `/v3/users/55df034b429644f2bc2ccd0f9f613f37` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:04.608123 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4453001c-f1d1-4b10-8fd9-7dccec980074 admin admin] RBAC: Authorizing `identity:delete_user(user_id=55df034b429644f2bc2ccd0f9f613f37)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:04.611552 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4453001c-f1d1-4b10-8fd9-7dccec980074 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:04.695598 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-4453001c-f1d1-4b10-8fd9-7dccec980074 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '55df034b429644f2bc2ccd0f9f613f37'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:04.708157 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-4453001c-f1d1-4b10-8fd9-7dccec980074 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '55df034b429644f2bc2ccd0f9f613f37'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:04.708931 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-4453001c-f1d1-4b10-8fd9-7dccec980074 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '55df034b429644f2bc2ccd0f9f613f37'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:04.709740 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-4453001c-f1d1-4b10-8fd9-7dccec980074 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '55df034b429644f2bc2ccd0f9f613f37'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:04.727533 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-4453001c-f1d1-4b10-8fd9-7dccec980074 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '55df034b429644f2bc2ccd0f9f613f37'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:04.754340 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1262/2531] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:03:04 2022] DELETE /identity/v3/users/55df034b429644f2bc2ccd0f9f613f37 => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:03:05.001419 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a82d5d6-9b96-4817-beac-6e8238de4163 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:05.003707 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-2a82d5d6-9b96-4817-beac-6e8238de4163 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:05.031110 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a82d5d6-9b96-4817-beac-6e8238de4163 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:05.117640 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc92055-5d95-40e8-962d-e3659bf3e443 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:05.118571 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc92055-5d95-40e8-962d-e3659bf3e443 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:05.118900 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc92055-5d95-40e8-962d-e3659bf3e443 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:05.119197 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc92055-5d95-40e8-962d-e3659bf3e443 admin admin] PATH_INFO: `/v3/projects/60700a7ed187472bb91e8e138940f65d` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:05.120600 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc92055-5d95-40e8-962d-e3659bf3e443 admin admin] RBAC: Authorizing `identity:delete_project(project_id=60700a7ed187472bb91e8e138940f65d)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:05.122685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc92055-5d95-40e8-962d-e3659bf3e443 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:05.177425 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-bdc92055-5d95-40e8-962d-e3659bf3e443 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '60700a7ed187472bb91e8e138940f65d', 'request_id': 'req-bdc92055-5d95-40e8-962d-e3659bf3e443'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:05.179038 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:05.186060 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-bdc92055-5d95-40e8-962d-e3659bf3e443 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '60700a7ed187472bb91e8e138940f65d', 'request_id': 'req-bdc92055-5d95-40e8-962d-e3659bf3e443'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:05.197017 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-bdc92055-5d95-40e8-962d-e3659bf3e443 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '60700a7ed187472bb91e8e138940f65d', 'request_id': 'req-bdc92055-5d95-40e8-962d-e3659bf3e443'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:05.218397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-bdc92055-5d95-40e8-962d-e3659bf3e443 admin admin] The token cache is being invalidate because project 60700a7ed187472bb91e8e138940f65d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:03:05.218928 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-bdc92055-5d95-40e8-962d-e3659bf3e443 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:05.220853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-bdc92055-5d95-40e8-962d-e3659bf3e443 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:05.224768 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1270/2532] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:03:05 2022] DELETE /identity/v3/projects/60700a7ed187472bb91e8e138940f65d => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:03:05.768784 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4453001c-f1d1-4b10-8fd9-7dccec980074 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:05.773170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-4453001c-f1d1-4b10-8fd9-7dccec980074 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:05.789263 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4453001c-f1d1-4b10-8fd9-7dccec980074 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:05.865239 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a747ac53-e320-4262-9bef-9bb70bd5b81f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:05.865889 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a747ac53-e320-4262-9bef-9bb70bd5b81f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:05.866166 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a747ac53-e320-4262-9bef-9bb70bd5b81f admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:05.866439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a747ac53-e320-4262-9bef-9bb70bd5b81f admin admin] PATH_INFO: `/v3/users/2f54379006454d6da18c0e6d1682b9c6` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:05.867699 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a747ac53-e320-4262-9bef-9bb70bd5b81f admin admin] RBAC: Authorizing `identity:delete_user(user_id=2f54379006454d6da18c0e6d1682b9c6)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:05.869828 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a747ac53-e320-4262-9bef-9bb70bd5b81f admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:05.930776 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-a747ac53-e320-4262-9bef-9bb70bd5b81f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2f54379006454d6da18c0e6d1682b9c6'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:05.943494 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-a747ac53-e320-4262-9bef-9bb70bd5b81f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2f54379006454d6da18c0e6d1682b9c6'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:05.944193 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-a747ac53-e320-4262-9bef-9bb70bd5b81f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2f54379006454d6da18c0e6d1682b9c6'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:05.944922 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-a747ac53-e320-4262-9bef-9bb70bd5b81f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2f54379006454d6da18c0e6d1682b9c6'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:05.958560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-a747ac53-e320-4262-9bef-9bb70bd5b81f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2f54379006454d6da18c0e6d1682b9c6'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:05.969089 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:05.986912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1263/2533] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:03:05 2022] DELETE /identity/v3/users/2f54379006454d6da18c0e6d1682b9c6 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:03:05.991785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b658338c-91ee-4e43-b2ac-84fb2d419884 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:05.992062 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b658338c-91ee-4e43-b2ac-84fb2d419884 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:05.992409 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b658338c-91ee-4e43-b2ac-84fb2d419884 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:06.014075 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-b658338c-91ee-4e43-b2ac-84fb2d419884 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:06.015325 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-b658338c-91ee-4e43-b2ac-84fb2d419884 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:06.101373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1271/2534] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:03:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:03:06.112646 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a747ac53-e320-4262-9bef-9bb70bd5b81f admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:06.114081 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-a747ac53-e320-4262-9bef-9bb70bd5b81f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:06.200026 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b658338c-91ee-4e43-b2ac-84fb2d419884 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:06.200532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a747ac53-e320-4262-9bef-9bb70bd5b81f admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:06.201547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-b658338c-91ee-4e43-b2ac-84fb2d419884 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:06.217278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f0b294-0746-4750-98ea-75d6631c099a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:06.219233 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f0b294-0746-4750-98ea-75d6631c099a service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:06.220288 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f0b294-0746-4750-98ea-75d6631c099a service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:06.220288 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f0b294-0746-4750-98ea-75d6631c099a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:06.253343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f0b294-0746-4750-98ea-75d6631c099a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:06.255208 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f0b294-0746-4750-98ea-75d6631c099a service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:06.271066 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1264/2535] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:03:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:03:06.348793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b658338c-91ee-4e43-b2ac-84fb2d419884 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:06.361662 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cedf02-3843-4e3e-8a19-ec81ba0ee528 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:06.362498 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cedf02-3843-4e3e-8a19-ec81ba0ee528 service nova] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:06.362843 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cedf02-3843-4e3e-8a19-ec81ba0ee528 service nova] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:06.363250 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cedf02-3843-4e3e-8a19-ec81ba0ee528 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:06.366504 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-b3cedf02-3843-4e3e-8a19-ec81ba0ee528 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:06.392499 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cedf02-3843-4e3e-8a19-ec81ba0ee528 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:06.394303 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cedf02-3843-4e3e-8a19-ec81ba0ee528 service nova] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:06.424165 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1272/2536] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:03:06 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:03:06.664834 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f0b294-0746-4750-98ea-75d6631c099a service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:06.666329 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d7f0b294-0746-4750-98ea-75d6631c099a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:06.681886 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f0b294-0746-4750-98ea-75d6631c099a service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:06.688286 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9943fc01-8fb4-40ac-8956-f8c7511e6dde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:06.689222 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9943fc01-8fb4-40ac-8956-f8c7511e6dde admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:06.689807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9943fc01-8fb4-40ac-8956-f8c7511e6dde admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:06.690697 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9943fc01-8fb4-40ac-8956-f8c7511e6dde admin admin] PATH_INFO: `/v3/projects/de7c0f943cc14eb38b908863f391241a` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:06.691930 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9943fc01-8fb4-40ac-8956-f8c7511e6dde admin admin] RBAC: Authorizing `identity:delete_project(project_id=de7c0f943cc14eb38b908863f391241a)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:06.695892 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9943fc01-8fb4-40ac-8956-f8c7511e6dde admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:06.775953 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9943fc01-8fb4-40ac-8956-f8c7511e6dde admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'de7c0f943cc14eb38b908863f391241a', 'request_id': 'req-9943fc01-8fb4-40ac-8956-f8c7511e6dde'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:06.787870 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9943fc01-8fb4-40ac-8956-f8c7511e6dde admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'de7c0f943cc14eb38b908863f391241a', 'request_id': 'req-9943fc01-8fb4-40ac-8956-f8c7511e6dde'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:06.798849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9943fc01-8fb4-40ac-8956-f8c7511e6dde admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'de7c0f943cc14eb38b908863f391241a', 'request_id': 'req-9943fc01-8fb4-40ac-8956-f8c7511e6dde'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:06.819175 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9943fc01-8fb4-40ac-8956-f8c7511e6dde admin admin] The token cache is being invalidate because project de7c0f943cc14eb38b908863f391241a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:03:06.819515 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9943fc01-8fb4-40ac-8956-f8c7511e6dde admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:06.820278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9943fc01-8fb4-40ac-8956-f8c7511e6dde admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:06.822078 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1265/2537] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:03:06 2022] DELETE /identity/v3/projects/de7c0f943cc14eb38b908863f391241a => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:03:06.827377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cedf02-3843-4e3e-8a19-ec81ba0ee528 service nova] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:06.839795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cedf02-3843-4e3e-8a19-ec81ba0ee528 service nova] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:06.910286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca95c9d2-690f-4ef3-ad7b-c25695f86b43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:06.910977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca95c9d2-690f-4ef3-ad7b-c25695f86b43 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:06.911257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca95c9d2-690f-4ef3-ad7b-c25695f86b43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:06.911530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca95c9d2-690f-4ef3-ad7b-c25695f86b43 admin admin] PATH_INFO: `/v3/users/e879c9752f15473cbbfabee5c21decc9` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:06.912759 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca95c9d2-690f-4ef3-ad7b-c25695f86b43 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e879c9752f15473cbbfabee5c21decc9)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:06.915580 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca95c9d2-690f-4ef3-ad7b-c25695f86b43 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:06.921529 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:DELETE=3,keystone:SELECT=96,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:06.975150 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-ca95c9d2-690f-4ef3-ad7b-c25695f86b43 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e879c9752f15473cbbfabee5c21decc9'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:06.983808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-ca95c9d2-690f-4ef3-ad7b-c25695f86b43 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e879c9752f15473cbbfabee5c21decc9'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:06.998982 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-ca95c9d2-690f-4ef3-ad7b-c25695f86b43 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e879c9752f15473cbbfabee5c21decc9'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:06.999844 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-ca95c9d2-690f-4ef3-ad7b-c25695f86b43 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e879c9752f15473cbbfabee5c21decc9'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:07.007280 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-ca95c9d2-690f-4ef3-ad7b-c25695f86b43 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e879c9752f15473cbbfabee5c21decc9'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:07.013256 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1273/2538] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:03:06 2022] DELETE /identity/v3/users/e879c9752f15473cbbfabee5c21decc9 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:03:07.334272 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9943fc01-8fb4-40ac-8956-f8c7511e6dde admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:07.336713 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-9943fc01-8fb4-40ac-8956-f8c7511e6dde admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:07.355728 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9943fc01-8fb4-40ac-8956-f8c7511e6dde admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:07.429513 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0bbed1-d750-4794-a97d-e2f07b8a6271 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:07.432748 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0bbed1-d750-4794-a97d-e2f07b8a6271 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:07.432748 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0bbed1-d750-4794-a97d-e2f07b8a6271 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:07.432748 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0bbed1-d750-4794-a97d-e2f07b8a6271 admin admin] PATH_INFO: `/v3/projects/5d944a07614f4a94a84194803db359bd` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:07.433569 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0bbed1-d750-4794-a97d-e2f07b8a6271 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5d944a07614f4a94a84194803db359bd)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:07.435867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0bbed1-d750-4794-a97d-e2f07b8a6271 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:07.465401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:07.502060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-5e0bbed1-d750-4794-a97d-e2f07b8a6271 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5d944a07614f4a94a84194803db359bd', 'request_id': 'req-5e0bbed1-d750-4794-a97d-e2f07b8a6271'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:07.507646 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-5e0bbed1-d750-4794-a97d-e2f07b8a6271 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5d944a07614f4a94a84194803db359bd', 'request_id': 'req-5e0bbed1-d750-4794-a97d-e2f07b8a6271'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:07.523199 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-5e0bbed1-d750-4794-a97d-e2f07b8a6271 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5d944a07614f4a94a84194803db359bd', 'request_id': 'req-5e0bbed1-d750-4794-a97d-e2f07b8a6271'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:07.537440 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-5e0bbed1-d750-4794-a97d-e2f07b8a6271 admin admin] The token cache is being invalidate because project 5d944a07614f4a94a84194803db359bd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:03:07.537889 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-5e0bbed1-d750-4794-a97d-e2f07b8a6271 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:07.538609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-5e0bbed1-d750-4794-a97d-e2f07b8a6271 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:07.540556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1266/2539] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:03:07 2022] DELETE /identity/v3/projects/5d944a07614f4a94a84194803db359bd => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:03:07.545117 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ad6a05-9abf-416b-bf7c-887ed7a85fc4 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:07.545605 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ad6a05-9abf-416b-bf7c-887ed7a85fc4 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:07.545803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ad6a05-9abf-416b-bf7c-887ed7a85fc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:07.567167 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-94ad6a05-9abf-416b-bf7c-887ed7a85fc4 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:07.568496 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-94ad6a05-9abf-416b-bf7c-887ed7a85fc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:07.642150 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1274/2540] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:03:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:03:07.651225 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919ea43b-c2e6-43a8-8444-2137e8672be9 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:07.651511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919ea43b-c2e6-43a8-8444-2137e8672be9 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:07.651794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919ea43b-c2e6-43a8-8444-2137e8672be9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:07.677863 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-919ea43b-c2e6-43a8-8444-2137e8672be9 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:07.679164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-919ea43b-c2e6-43a8-8444-2137e8672be9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:07.689577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1267/2541] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:03:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:03:07.695623 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ad6a05-9abf-416b-bf7c-887ed7a85fc4 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:07.703710 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ad6a05-9abf-416b-bf7c-887ed7a85fc4 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:07.708697 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7d9697-c20e-48f7-a0e4-aa2e9708310d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:07.709310 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7d9697-c20e-48f7-a0e4-aa2e9708310d admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:07.709593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7d9697-c20e-48f7-a0e4-aa2e9708310d admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:07.709870 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7d9697-c20e-48f7-a0e4-aa2e9708310d admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:07.710622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7d9697-c20e-48f7-a0e4-aa2e9708310d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:03:07.710968 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7d9697-c20e-48f7-a0e4-aa2e9708310d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:07.712646 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7d9697-c20e-48f7-a0e4-aa2e9708310d admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:07.727012 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1275/2542] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:03:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 01:03:07.730406 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919ea43b-c2e6-43a8-8444-2137e8672be9 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:07.740596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919ea43b-c2e6-43a8-8444-2137e8672be9 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:07.746402 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a75e89-df19-4f22-aebd-4c7bbae8fb48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:07.747050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a75e89-df19-4f22-aebd-4c7bbae8fb48 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:07.747331 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a75e89-df19-4f22-aebd-4c7bbae8fb48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:07.747604 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a75e89-df19-4f22-aebd-4c7bbae8fb48 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:07.748252 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a75e89-df19-4f22-aebd-4c7bbae8fb48 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:07.751230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a75e89-df19-4f22-aebd-4c7bbae8fb48 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:07.779793 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1268/2543] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:03:07 2022] POST /identity/v3/projects => generated 383 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:03:07.784427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7d9697-c20e-48f7-a0e4-aa2e9708310d admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:07.798533 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7d9697-c20e-48f7-a0e4-aa2e9708310d admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:07.893843 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33ebe2c-c2dc-41c3-b2b8-8cd98dc65200 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:07.894686 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33ebe2c-c2dc-41c3-b2b8-8cd98dc65200 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:07.895095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33ebe2c-c2dc-41c3-b2b8-8cd98dc65200 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:07.895490 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33ebe2c-c2dc-41c3-b2b8-8cd98dc65200 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:07.896240 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33ebe2c-c2dc-41c3-b2b8-8cd98dc65200 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:07.899182 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33ebe2c-c2dc-41c3-b2b8-8cd98dc65200 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:07.923410 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:07.932010 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1276/2544] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:03:07 2022] POST /identity/v3/users => generated 292 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 01:03:07.935819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a75e89-df19-4f22-aebd-4c7bbae8fb48 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:07.951001 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a75e89-df19-4f22-aebd-4c7bbae8fb48 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:07.956057 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db1b2a8-4d08-409f-91b5-ec298f9d6ba4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:07.957749 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db1b2a8-4d08-409f-91b5-ec298f9d6ba4 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:07.958137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db1b2a8-4d08-409f-91b5-ec298f9d6ba4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:07.958684 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db1b2a8-4d08-409f-91b5-ec298f9d6ba4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:07.959466 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db1b2a8-4d08-409f-91b5-ec298f9d6ba4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:07.964344 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db1b2a8-4d08-409f-91b5-ec298f9d6ba4 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:07.974519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1269/2545] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:03:07 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:03:07.978951 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33ebe2c-c2dc-41c3-b2b8-8cd98dc65200 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:07.991214 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33ebe2c-c2dc-41c3-b2b8-8cd98dc65200 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:07.996559 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b77f928-70eb-4cbe-b285-b5b3a5431262 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:07.997635 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b77f928-70eb-4cbe-b285-b5b3a5431262 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:07.998069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b77f928-70eb-4cbe-b285-b5b3a5431262 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:07.998525 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b77f928-70eb-4cbe-b285-b5b3a5431262 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:07.999350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b77f928-70eb-4cbe-b285-b5b3a5431262 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:08.001183 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b77f928-70eb-4cbe-b285-b5b3a5431262 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:08.013634 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1277/2546] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:03:07 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:03:08.017750 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db1b2a8-4d08-409f-91b5-ec298f9d6ba4 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:08.044900 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db1b2a8-4d08-409f-91b5-ec298f9d6ba4 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:08.050231 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24a80f0-0aee-4d8b-8415-059086d019a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:08.051762 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24a80f0-0aee-4d8b-8415-059086d019a4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:08.052219 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24a80f0-0aee-4d8b-8415-059086d019a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:08.052679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24a80f0-0aee-4d8b-8415-059086d019a4 admin admin] PATH_INFO: `/v3/projects/7fd2c0a470504ee5b7b5603759922920/users/db59ac2fdf8d4be9a909835854534810/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:08.075147 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24a80f0-0aee-4d8b-8415-059086d019a4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7fd2c0a470504ee5b7b5603759922920, user_id=db59ac2fdf8d4be9a909835854534810, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:08.078061 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24a80f0-0aee-4d8b-8415-059086d019a4 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:08.094760 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1270/2547] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:03:08 2022] PUT /identity/v3/projects/7fd2c0a470504ee5b7b5603759922920/users/db59ac2fdf8d4be9a909835854534810/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:03:08.105652 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed70574-cb10-4736-a8de-e9ad59fc90f6 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:08.105942 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed70574-cb10-4736-a8de-e9ad59fc90f6 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:08.106239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed70574-cb10-4736-a8de-e9ad59fc90f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:08.142095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-eed70574-cb10-4736-a8de-e9ad59fc90f6 None None] MFA Rules not processed for user `db59ac2fdf8d4be9a909835854534810`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:08.143346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-eed70574-cb10-4736-a8de-e9ad59fc90f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:08.223365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1278/2548] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:03:08 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:03:08.228440 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24a80f0-0aee-4d8b-8415-059086d019a4 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:08.306766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24a80f0-0aee-4d8b-8415-059086d019a4 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:08.312369 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ce6d36-8159-4c41-8fa3-9d7867d9520e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:08.312880 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ce6d36-8159-4c41-8fa3-9d7867d9520e admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:08.312880 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ce6d36-8159-4c41-8fa3-9d7867d9520e admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:08.313312 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ce6d36-8159-4c41-8fa3-9d7867d9520e admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:08.314272 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ce6d36-8159-4c41-8fa3-9d7867d9520e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:08.316346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ce6d36-8159-4c41-8fa3-9d7867d9520e admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:08.356877 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1271/2549] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:03:08 2022] POST /identity/v3/projects => generated 381 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:03:08.358054 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed70574-cb10-4736-a8de-e9ad59fc90f6 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:08.373040 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed70574-cb10-4736-a8de-e9ad59fc90f6 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:08.453107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9999cebe-7f39-4eed-9a34-cae011812038 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:08.453848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9999cebe-7f39-4eed-9a34-cae011812038 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:08.454186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9999cebe-7f39-4eed-9a34-cae011812038 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:08.454556 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9999cebe-7f39-4eed-9a34-cae011812038 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:08.455201 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9999cebe-7f39-4eed-9a34-cae011812038 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:08.457585 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9999cebe-7f39-4eed-9a34-cae011812038 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:08.482936 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1279/2550] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:03:08 2022] POST /identity/v3/users => generated 297 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 01:03:08.486923 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ce6d36-8159-4c41-8fa3-9d7867d9520e admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:08.505056 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ce6d36-8159-4c41-8fa3-9d7867d9520e admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:08.515853 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30c0738-693b-4264-abeb-3aff8b76c663 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:08.516821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30c0738-693b-4264-abeb-3aff8b76c663 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:08.517331 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30c0738-693b-4264-abeb-3aff8b76c663 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:08.517751 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30c0738-693b-4264-abeb-3aff8b76c663 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:08.519106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30c0738-693b-4264-abeb-3aff8b76c663 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:08.520972 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30c0738-693b-4264-abeb-3aff8b76c663 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:08.535549 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1272/2551] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:03:08 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 01:03:08.539855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9999cebe-7f39-4eed-9a34-cae011812038 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:08.551034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9999cebe-7f39-4eed-9a34-cae011812038 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:08.556280 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f139f1-871f-46ea-89e4-903eb4fa5e91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:08.557107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f139f1-871f-46ea-89e4-903eb4fa5e91 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:08.557404 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f139f1-871f-46ea-89e4-903eb4fa5e91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:08.557739 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f139f1-871f-46ea-89e4-903eb4fa5e91 admin admin] PATH_INFO: `/v3/projects/570e98d10b684272b72e96d9b0c1831c/users/db77028856d1477f8151fcb9df8449af/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:08.575539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f139f1-871f-46ea-89e4-903eb4fa5e91 admin admin] RBAC: Authorizing `identity:create_grant(project_id=570e98d10b684272b72e96d9b0c1831c, user_id=db77028856d1477f8151fcb9df8449af, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:08.579322 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f139f1-871f-46ea-89e4-903eb4fa5e91 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:08.597710 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1280/2552] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:03:08 2022] PUT /identity/v3/projects/570e98d10b684272b72e96d9b0c1831c/users/db77028856d1477f8151fcb9df8449af/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:03:08.600147 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30c0738-693b-4264-abeb-3aff8b76c663 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:08.614258 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30c0738-693b-4264-abeb-3aff8b76c663 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:08.631311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:08.692760 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2992eb6-c8f7-4f31-9d72-c8f3542d3a24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:08.693624 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2992eb6-c8f7-4f31-9d72-c8f3542d3a24 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:08.694004 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2992eb6-c8f7-4f31-9d72-c8f3542d3a24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:08.694380 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2992eb6-c8f7-4f31-9d72-c8f3542d3a24 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:08.695138 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2992eb6-c8f7-4f31-9d72-c8f3542d3a24 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:08.696873 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2992eb6-c8f7-4f31-9d72-c8f3542d3a24 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:08.710428 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1273/2553] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:03:08 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:03:08.714944 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f139f1-871f-46ea-89e4-903eb4fa5e91 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:08.726322 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f139f1-871f-46ea-89e4-903eb4fa5e91 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:08.732067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f46696-7e8d-4cd2-8cc7-bd47d976a9a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:08.732731 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f46696-7e8d-4cd2-8cc7-bd47d976a9a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:08.733033 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f46696-7e8d-4cd2-8cc7-bd47d976a9a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:08.733306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f46696-7e8d-4cd2-8cc7-bd47d976a9a5 admin admin] PATH_INFO: `/v3/projects/570e98d10b684272b72e96d9b0c1831c/users/db77028856d1477f8151fcb9df8449af/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:08.735262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f46696-7e8d-4cd2-8cc7-bd47d976a9a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=570e98d10b684272b72e96d9b0c1831c, user_id=db77028856d1477f8151fcb9df8449af, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:08.737890 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f46696-7e8d-4cd2-8cc7-bd47d976a9a5 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:08.749863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1281/2554] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:03:08 2022] PUT /identity/v3/projects/570e98d10b684272b72e96d9b0c1831c/users/db77028856d1477f8151fcb9df8449af/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:03:08.760087 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688bc1b8-87a7-4fe0-b969-e69545192726 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:08.760560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688bc1b8-87a7-4fe0-b969-e69545192726 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:08.761558 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688bc1b8-87a7-4fe0-b969-e69545192726 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:08.796343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-688bc1b8-87a7-4fe0-b969-e69545192726 None None] MFA Rules not processed for user `db77028856d1477f8151fcb9df8449af`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:08.797695 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-688bc1b8-87a7-4fe0-b969-e69545192726 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:08.895564 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1274/2555] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:03:08 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:03:08.904165 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f46696-7e8d-4cd2-8cc7-bd47d976a9a5 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:08.906077 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-88f46696-7e8d-4cd2-8cc7-bd47d976a9a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:08.955800 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:08.986611 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f46696-7e8d-4cd2-8cc7-bd47d976a9a5 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:08.990360 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44d5e48-b24d-430c-9310-19e2f6970efa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:08.991037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44d5e48-b24d-430c-9310-19e2f6970efa service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:08.991314 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44d5e48-b24d-430c-9310-19e2f6970efa service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:08.991614 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44d5e48-b24d-430c-9310-19e2f6970efa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:09.003783 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44d5e48-b24d-430c-9310-19e2f6970efa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:09.005381 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44d5e48-b24d-430c-9310-19e2f6970efa service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:09.017105 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1282/2556] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:03:08 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:03:09.142399 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688bc1b8-87a7-4fe0-b969-e69545192726 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:09.157298 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688bc1b8-87a7-4fe0-b969-e69545192726 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:09.161347 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e72368-692d-4d0c-94d6-a99afb570584 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:09.162210 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e72368-692d-4d0c-94d6-a99afb570584 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:09.162594 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e72368-692d-4d0c-94d6-a99afb570584 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:09.162964 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e72368-692d-4d0c-94d6-a99afb570584 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:09.238956 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e72368-692d-4d0c-94d6-a99afb570584 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:09.240905 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e72368-692d-4d0c-94d6-a99afb570584 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:09.261946 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1275/2557] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:03:09 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:03:09.582319 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44d5e48-b24d-430c-9310-19e2f6970efa service cinder] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:09.659042 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44d5e48-b24d-430c-9310-19e2f6970efa service cinder] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:09.666990 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe79d75-19ac-4354-9015-eb789b7a1b1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:09.667838 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe79d75-19ac-4354-9015-eb789b7a1b1b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:09.668339 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe79d75-19ac-4354-9015-eb789b7a1b1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:09.668643 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe79d75-19ac-4354-9015-eb789b7a1b1b admin admin] PATH_INFO: `/v3/users/db59ac2fdf8d4be9a909835854534810` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:09.669877 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe79d75-19ac-4354-9015-eb789b7a1b1b admin admin] RBAC: Authorizing `identity:delete_user(user_id=db59ac2fdf8d4be9a909835854534810)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:09.672011 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe79d75-19ac-4354-9015-eb789b7a1b1b admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:09.733044 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-bbe79d75-19ac-4354-9015-eb789b7a1b1b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'db59ac2fdf8d4be9a909835854534810'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:09.741664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-bbe79d75-19ac-4354-9015-eb789b7a1b1b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'db59ac2fdf8d4be9a909835854534810'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:09.756517 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-bbe79d75-19ac-4354-9015-eb789b7a1b1b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'db59ac2fdf8d4be9a909835854534810'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:09.757270 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-bbe79d75-19ac-4354-9015-eb789b7a1b1b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'db59ac2fdf8d4be9a909835854534810'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:09.766300 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-bbe79d75-19ac-4354-9015-eb789b7a1b1b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'db59ac2fdf8d4be9a909835854534810'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:09.773249 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1283/2558] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:03:09 2022] DELETE /identity/v3/users/db59ac2fdf8d4be9a909835854534810 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:03:09.778304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bcdf7f-91e8-45a2-a10b-d8a00b9c96d9 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:09.778654 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bcdf7f-91e8-45a2-a10b-d8a00b9c96d9 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:09.778932 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bcdf7f-91e8-45a2-a10b-d8a00b9c96d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:09.800090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-b3bcdf7f-91e8-45a2-a10b-d8a00b9c96d9 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:09.801347 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-b3bcdf7f-91e8-45a2-a10b-d8a00b9c96d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:09.837827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:09.870514 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1276/2559] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:03:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:03:09.880840 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe79d75-19ac-4354-9015-eb789b7a1b1b admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:09.882623 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-bbe79d75-19ac-4354-9015-eb789b7a1b1b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:09.946023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe79d75-19ac-4354-9015-eb789b7a1b1b admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:09.949377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc697ae-b35b-42cf-8843-129428138cb4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:09.950008 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc697ae-b35b-42cf-8843-129428138cb4 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:09.950291 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc697ae-b35b-42cf-8843-129428138cb4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:09.950594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc697ae-b35b-42cf-8843-129428138cb4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:09.960711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:09.963637 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc697ae-b35b-42cf-8843-129428138cb4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:09.965549 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc697ae-b35b-42cf-8843-129428138cb4 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:09.979113 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1284/2560] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:03:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:03:10.331782 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bcdf7f-91e8-45a2-a10b-d8a00b9c96d9 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:10.333615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-b3bcdf7f-91e8-45a2-a10b-d8a00b9c96d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:10.363120 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bcdf7f-91e8-45a2-a10b-d8a00b9c96d9 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:10.369842 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cc6a24-694d-4357-ae4c-29f792a9dbfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:10.370744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cc6a24-694d-4357-ae4c-29f792a9dbfc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:10.370744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cc6a24-694d-4357-ae4c-29f792a9dbfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:10.371236 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cc6a24-694d-4357-ae4c-29f792a9dbfc admin admin] PATH_INFO: `/v3/projects/7fd2c0a470504ee5b7b5603759922920` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:10.372820 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43cc6a24-694d-4357-ae4c-29f792a9dbfc admin admin] RBAC: Authorizing `identity:delete_project(project_id=7fd2c0a470504ee5b7b5603759922920)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:10.375468 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43cc6a24-694d-4357-ae4c-29f792a9dbfc admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:10.446625 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-43cc6a24-694d-4357-ae4c-29f792a9dbfc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7fd2c0a470504ee5b7b5603759922920', 'request_id': 'req-43cc6a24-694d-4357-ae4c-29f792a9dbfc'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:10.453707 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-43cc6a24-694d-4357-ae4c-29f792a9dbfc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7fd2c0a470504ee5b7b5603759922920', 'request_id': 'req-43cc6a24-694d-4357-ae4c-29f792a9dbfc'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:10.464149 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-43cc6a24-694d-4357-ae4c-29f792a9dbfc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7fd2c0a470504ee5b7b5603759922920', 'request_id': 'req-43cc6a24-694d-4357-ae4c-29f792a9dbfc'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:10.488956 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-43cc6a24-694d-4357-ae4c-29f792a9dbfc admin admin] The token cache is being invalidate because project 7fd2c0a470504ee5b7b5603759922920 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:03:10.489158 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-43cc6a24-694d-4357-ae4c-29f792a9dbfc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:10.491055 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-43cc6a24-694d-4357-ae4c-29f792a9dbfc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:10.496787 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1277/2561] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:03:10 2022] DELETE /identity/v3/projects/7fd2c0a470504ee5b7b5603759922920 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:03:10.497058 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc697ae-b35b-42cf-8843-129428138cb4 service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:10.511988 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc697ae-b35b-42cf-8843-129428138cb4 service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:10.582806 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b447bfd3-22d0-4953-9221-580dc3bb5f28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:10.583457 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b447bfd3-22d0-4953-9221-580dc3bb5f28 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:10.583726 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b447bfd3-22d0-4953-9221-580dc3bb5f28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:10.584000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b447bfd3-22d0-4953-9221-580dc3bb5f28 admin admin] PATH_INFO: `/v3/users/db77028856d1477f8151fcb9df8449af` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:10.585133 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b447bfd3-22d0-4953-9221-580dc3bb5f28 admin admin] RBAC: Authorizing `identity:delete_user(user_id=db77028856d1477f8151fcb9df8449af)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:10.587726 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b447bfd3-22d0-4953-9221-580dc3bb5f28 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:10.641890 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-b447bfd3-22d0-4953-9221-580dc3bb5f28 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'db77028856d1477f8151fcb9df8449af'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:10.651498 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-b447bfd3-22d0-4953-9221-580dc3bb5f28 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'db77028856d1477f8151fcb9df8449af'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:10.665293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-b447bfd3-22d0-4953-9221-580dc3bb5f28 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'db77028856d1477f8151fcb9df8449af'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:10.665999 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-b447bfd3-22d0-4953-9221-580dc3bb5f28 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'db77028856d1477f8151fcb9df8449af'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:10.675387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-b447bfd3-22d0-4953-9221-580dc3bb5f28 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'db77028856d1477f8151fcb9df8449af'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:10.681955 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1285/2562] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:03:10 2022] DELETE /identity/v3/users/db77028856d1477f8151fcb9df8449af => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 01:03:10.993697 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cc6a24-694d-4357-ae4c-29f792a9dbfc admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:10.994210 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-43cc6a24-694d-4357-ae4c-29f792a9dbfc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:11.025346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cc6a24-694d-4357-ae4c-29f792a9dbfc admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:11.097852 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f544208-83d3-44ce-b5dd-51825b67fe68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:11.098634 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f544208-83d3-44ce-b5dd-51825b67fe68 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:11.099049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f544208-83d3-44ce-b5dd-51825b67fe68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:11.099495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f544208-83d3-44ce-b5dd-51825b67fe68 admin admin] PATH_INFO: `/v3/projects/570e98d10b684272b72e96d9b0c1831c` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:11.103767 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f544208-83d3-44ce-b5dd-51825b67fe68 admin admin] RBAC: Authorizing `identity:delete_project(project_id=570e98d10b684272b72e96d9b0c1831c)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:11.106904 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f544208-83d3-44ce-b5dd-51825b67fe68 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:11.171234 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8f544208-83d3-44ce-b5dd-51825b67fe68 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '570e98d10b684272b72e96d9b0c1831c', 'request_id': 'req-8f544208-83d3-44ce-b5dd-51825b67fe68'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:11.177766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8f544208-83d3-44ce-b5dd-51825b67fe68 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '570e98d10b684272b72e96d9b0c1831c', 'request_id': 'req-8f544208-83d3-44ce-b5dd-51825b67fe68'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:11.181902 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:11.189204 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8f544208-83d3-44ce-b5dd-51825b67fe68 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '570e98d10b684272b72e96d9b0c1831c', 'request_id': 'req-8f544208-83d3-44ce-b5dd-51825b67fe68'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:11.205544 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8f544208-83d3-44ce-b5dd-51825b67fe68 admin admin] The token cache is being invalidate because project 570e98d10b684272b72e96d9b0c1831c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:03:11.205861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8f544208-83d3-44ce-b5dd-51825b67fe68 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:11.206852 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-8f544208-83d3-44ce-b5dd-51825b67fe68 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:11.208512 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1278/2563] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:03:10 2022] DELETE /identity/v3/projects/570e98d10b684272b72e96d9b0c1831c => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:03:11.214041 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121c0b25-1fde-4a83-85d1-95fb82cfafbc None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:11.214439 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121c0b25-1fde-4a83-85d1-95fb82cfafbc None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:11.214812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121c0b25-1fde-4a83-85d1-95fb82cfafbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:11.249812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-121c0b25-1fde-4a83-85d1-95fb82cfafbc None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:11.251216 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-121c0b25-1fde-4a83-85d1-95fb82cfafbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:11.324144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1286/2564] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:03:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 01:03:11.390107 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472a03f5-d1bd-43ff-a374-ebfd5fd6aea3 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:11.390615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472a03f5-d1bd-43ff-a374-ebfd5fd6aea3 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:11.390958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472a03f5-d1bd-43ff-a374-ebfd5fd6aea3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:11.412385 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-472a03f5-d1bd-43ff-a374-ebfd5fd6aea3 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:11.413602 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-472a03f5-d1bd-43ff-a374-ebfd5fd6aea3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:11.422288 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1279/2565] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:03:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:03:11.426601 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121c0b25-1fde-4a83-85d1-95fb82cfafbc None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:11.437903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121c0b25-1fde-4a83-85d1-95fb82cfafbc None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:11.443177 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d379df-c201-47fa-9ff0-fe4d798201c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:11.443799 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d379df-c201-47fa-9ff0-fe4d798201c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:11.444147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d379df-c201-47fa-9ff0-fe4d798201c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:11.444357 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d379df-c201-47fa-9ff0-fe4d798201c7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:11.445154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d379df-c201-47fa-9ff0-fe4d798201c7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:03:11.446029 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d379df-c201-47fa-9ff0-fe4d798201c7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:11.447805 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d379df-c201-47fa-9ff0-fe4d798201c7 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:11.468265 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1287/2566] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:03:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 01:03:11.469673 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472a03f5-d1bd-43ff-a374-ebfd5fd6aea3 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:11.484662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472a03f5-d1bd-43ff-a374-ebfd5fd6aea3 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:11.490977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defa867e-d355-4bab-918b-77be02960bf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:11.491941 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defa867e-d355-4bab-918b-77be02960bf3 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:11.492597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defa867e-d355-4bab-918b-77be02960bf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:11.493084 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defa867e-d355-4bab-918b-77be02960bf3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:11.494295 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defa867e-d355-4bab-918b-77be02960bf3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:11.496442 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defa867e-d355-4bab-918b-77be02960bf3 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:11.543227 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1280/2567] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:03:11 2022] POST /identity/v3/projects => generated 373 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:03:11.544691 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d379df-c201-47fa-9ff0-fe4d798201c7 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:11.550026 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:11.557692 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d379df-c201-47fa-9ff0-fe4d798201c7 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:11.624583 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bf1083-9dbd-4886-a647-6ce8fefddd06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:11.625419 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bf1083-9dbd-4886-a647-6ce8fefddd06 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:11.625518 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bf1083-9dbd-4886-a647-6ce8fefddd06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:11.625779 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bf1083-9dbd-4886-a647-6ce8fefddd06 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:11.626430 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59bf1083-9dbd-4886-a647-6ce8fefddd06 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:11.630331 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59bf1083-9dbd-4886-a647-6ce8fefddd06 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:11.656293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1288/2568] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:03:11 2022] POST /identity/v3/users => generated 287 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 01:03:11.657253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defa867e-d355-4bab-918b-77be02960bf3 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:11.672071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defa867e-d355-4bab-918b-77be02960bf3 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:11.678178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe88ab1-42ba-45d3-964b-ad3660907ea1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:11.678853 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe88ab1-42ba-45d3-964b-ad3660907ea1 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:11.679301 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe88ab1-42ba-45d3-964b-ad3660907ea1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:11.680095 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe88ab1-42ba-45d3-964b-ad3660907ea1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:11.680777 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe88ab1-42ba-45d3-964b-ad3660907ea1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:11.682427 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe88ab1-42ba-45d3-964b-ad3660907ea1 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:11.704148 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bf1083-9dbd-4886-a647-6ce8fefddd06 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:11.704566 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1281/2569] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:03:11 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:03:11.716396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bf1083-9dbd-4886-a647-6ce8fefddd06 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:11.722525 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5f39d6-8316-4717-832b-be3eb38684b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:11.723159 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5f39d6-8316-4717-832b-be3eb38684b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:11.723476 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5f39d6-8316-4717-832b-be3eb38684b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:11.723747 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5f39d6-8316-4717-832b-be3eb38684b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:11.725184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e5f39d6-8316-4717-832b-be3eb38684b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:11.726905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e5f39d6-8316-4717-832b-be3eb38684b4 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:11.741927 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1289/2570] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:03:11 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:03:11.747246 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe88ab1-42ba-45d3-964b-ad3660907ea1 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:11.763219 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe88ab1-42ba-45d3-964b-ad3660907ea1 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:11.769671 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab5cb02-a331-4ace-8972-8554c3ae7b06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:11.779805 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab5cb02-a331-4ace-8972-8554c3ae7b06 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:11.780247 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab5cb02-a331-4ace-8972-8554c3ae7b06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:11.780632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab5cb02-a331-4ace-8972-8554c3ae7b06 admin admin] PATH_INFO: `/v3/projects/3d423665f98144c7a598149e05e78837/users/cb7852e06ead4a40b4442bc1907b5604/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:11.806914 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab5cb02-a331-4ace-8972-8554c3ae7b06 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3d423665f98144c7a598149e05e78837, user_id=cb7852e06ead4a40b4442bc1907b5604, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:11.811499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab5cb02-a331-4ace-8972-8554c3ae7b06 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:11.839099 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1282/2571] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:03:11 2022] PUT /identity/v3/projects/3d423665f98144c7a598149e05e78837/users/cb7852e06ead4a40b4442bc1907b5604/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:03:11.842709 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cf8fe5-d317-4db5-908b-76b1bc14cb57 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:11.843003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cf8fe5-d317-4db5-908b-76b1bc14cb57 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:11.843281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cf8fe5-d317-4db5-908b-76b1bc14cb57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:11.876184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-90cf8fe5-d317-4db5-908b-76b1bc14cb57 None None] MFA Rules not processed for user `cb7852e06ead4a40b4442bc1907b5604`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:11.877554 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-90cf8fe5-d317-4db5-908b-76b1bc14cb57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:11.966384 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1290/2572] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:03:11 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:03:11.974578 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab5cb02-a331-4ace-8972-8554c3ae7b06 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:11.976681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8ab5cb02-a331-4ace-8972-8554c3ae7b06 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:12.047077 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab5cb02-a331-4ace-8972-8554c3ae7b06 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:12.051429 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d5d1c8-7b10-471e-ab3c-32a717d74ef9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:12.052102 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d5d1c8-7b10-471e-ab3c-32a717d74ef9 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:12.052369 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d5d1c8-7b10-471e-ab3c-32a717d74ef9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:12.052669 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d5d1c8-7b10-471e-ab3c-32a717d74ef9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:12.067910 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d5d1c8-7b10-471e-ab3c-32a717d74ef9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:12.069590 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d5d1c8-7b10-471e-ab3c-32a717d74ef9 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:12.092663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1283/2573] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:03:11 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:03:15.240793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90cf8fe5-d317-4db5-908b-76b1bc14cb57 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:15.320211 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90cf8fe5-d317-4db5-908b-76b1bc14cb57 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:15.326433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb14aa9-3f70-4cde-a223-3b64c81da3fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:15.327107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb14aa9-3f70-4cde-a223-3b64c81da3fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:15.327390 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb14aa9-3f70-4cde-a223-3b64c81da3fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:15.327665 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb14aa9-3f70-4cde-a223-3b64c81da3fc admin admin] PATH_INFO: `/v3/users/cb7852e06ead4a40b4442bc1907b5604` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:15.328797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb14aa9-3f70-4cde-a223-3b64c81da3fc admin admin] RBAC: Authorizing `identity:delete_user(user_id=cb7852e06ead4a40b4442bc1907b5604)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:15.330768 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb14aa9-3f70-4cde-a223-3b64c81da3fc admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:15.344628 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:15.385572 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-fcb14aa9-3f70-4cde-a223-3b64c81da3fc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cb7852e06ead4a40b4442bc1907b5604'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:15.394233 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-fcb14aa9-3f70-4cde-a223-3b64c81da3fc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cb7852e06ead4a40b4442bc1907b5604'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:15.406785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-fcb14aa9-3f70-4cde-a223-3b64c81da3fc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cb7852e06ead4a40b4442bc1907b5604'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:15.408476 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-fcb14aa9-3f70-4cde-a223-3b64c81da3fc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cb7852e06ead4a40b4442bc1907b5604'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:15.414157 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-fcb14aa9-3f70-4cde-a223-3b64c81da3fc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cb7852e06ead4a40b4442bc1907b5604'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:15.420873 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1291/2574] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:03:15 2022] DELETE /identity/v3/users/cb7852e06ead4a40b4442bc1907b5604 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:03:15.425796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a819e2a-ae3d-4d69-9348-2de0ffd6e331 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:15.426272 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a819e2a-ae3d-4d69-9348-2de0ffd6e331 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:15.426646 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a819e2a-ae3d-4d69-9348-2de0ffd6e331 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:15.446636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-3a819e2a-ae3d-4d69-9348-2de0ffd6e331 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:15.447825 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-3a819e2a-ae3d-4d69-9348-2de0ffd6e331 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:15.508605 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1284/2575] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:03:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:03:15.518445 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb14aa9-3f70-4cde-a223-3b64c81da3fc admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:15.519914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-fcb14aa9-3f70-4cde-a223-3b64c81da3fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:15.577380 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb14aa9-3f70-4cde-a223-3b64c81da3fc admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:15.581004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21d6d84-60b6-467a-b245-e1cd165baa14 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:15.581874 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21d6d84-60b6-467a-b245-e1cd165baa14 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:15.582153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21d6d84-60b6-467a-b245-e1cd165baa14 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:15.582427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21d6d84-60b6-467a-b245-e1cd165baa14 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:15.592963 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21d6d84-60b6-467a-b245-e1cd165baa14 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:15.594392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21d6d84-60b6-467a-b245-e1cd165baa14 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:15.605166 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1292/2576] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:03:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:03:15.871891 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a819e2a-ae3d-4d69-9348-2de0ffd6e331 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:15.873511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-3a819e2a-ae3d-4d69-9348-2de0ffd6e331 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:15.886541 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a819e2a-ae3d-4d69-9348-2de0ffd6e331 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:15.891345 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bc4f78-4536-4923-8ea5-bede5d72d042 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:15.892062 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bc4f78-4536-4923-8ea5-bede5d72d042 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:15.892448 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bc4f78-4536-4923-8ea5-bede5d72d042 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:15.892791 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bc4f78-4536-4923-8ea5-bede5d72d042 admin admin] PATH_INFO: `/v3/projects/3d423665f98144c7a598149e05e78837` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:15.894000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bc4f78-4536-4923-8ea5-bede5d72d042 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3d423665f98144c7a598149e05e78837)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:15.895935 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bc4f78-4536-4923-8ea5-bede5d72d042 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:15.926916 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=2,keystone:DELETE=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:15.944913 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-30bc4f78-4536-4923-8ea5-bede5d72d042 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3d423665f98144c7a598149e05e78837', 'request_id': 'req-30bc4f78-4536-4923-8ea5-bede5d72d042'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:15.951023 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-30bc4f78-4536-4923-8ea5-bede5d72d042 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3d423665f98144c7a598149e05e78837', 'request_id': 'req-30bc4f78-4536-4923-8ea5-bede5d72d042'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:15.958952 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-30bc4f78-4536-4923-8ea5-bede5d72d042 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3d423665f98144c7a598149e05e78837', 'request_id': 'req-30bc4f78-4536-4923-8ea5-bede5d72d042'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:15.970347 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-30bc4f78-4536-4923-8ea5-bede5d72d042 admin admin] The token cache is being invalidate because project 3d423665f98144c7a598149e05e78837 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:03:15.970670 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-30bc4f78-4536-4923-8ea5-bede5d72d042 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:15.971336 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-30bc4f78-4536-4923-8ea5-bede5d72d042 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:15.972919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1285/2577] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:03:15 2022] DELETE /identity/v3/projects/3d423665f98144c7a598149e05e78837 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:03:15.978515 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c4c2cb-59bd-4750-9108-114a450c3531 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:15.978784 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c4c2cb-59bd-4750-9108-114a450c3531 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:15.979057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c4c2cb-59bd-4750-9108-114a450c3531 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:15.998974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-b4c4c2cb-59bd-4750-9108-114a450c3531 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:16.000172 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-b4c4c2cb-59bd-4750-9108-114a450c3531 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:16.059021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1293/2578] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:03:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:03:16.067135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc583db-aa1a-4213-9cb9-8f469ba9d619 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:16.067434 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc583db-aa1a-4213-9cb9-8f469ba9d619 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:16.067753 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc583db-aa1a-4213-9cb9-8f469ba9d619 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:16.088360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-3bc583db-aa1a-4213-9cb9-8f469ba9d619 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:16.089575 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-3bc583db-aa1a-4213-9cb9-8f469ba9d619 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:16.097585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1286/2579] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:03:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:03:16.103996 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c4c2cb-59bd-4750-9108-114a450c3531 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:16.114188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c4c2cb-59bd-4750-9108-114a450c3531 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:16.118797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0f9d2f-f94a-4125-9183-567ae817be90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:16.119397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0f9d2f-f94a-4125-9183-567ae817be90 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:16.119674 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0f9d2f-f94a-4125-9183-567ae817be90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:16.119944 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0f9d2f-f94a-4125-9183-567ae817be90 admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:16.120583 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0f9d2f-f94a-4125-9183-567ae817be90 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:03:16.120886 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0f9d2f-f94a-4125-9183-567ae817be90 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:16.122489 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0f9d2f-f94a-4125-9183-567ae817be90 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:16.133791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1294/2580] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:03:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 01:03:16.138449 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc583db-aa1a-4213-9cb9-8f469ba9d619 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:16.147518 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc583db-aa1a-4213-9cb9-8f469ba9d619 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:16.152672 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8c7152-824d-4b46-872c-29c108581721 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:16.153262 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8c7152-824d-4b46-872c-29c108581721 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:16.153539 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8c7152-824d-4b46-872c-29c108581721 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:16.153807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8c7152-824d-4b46-872c-29c108581721 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:16.154436 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8c7152-824d-4b46-872c-29c108581721 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:16.156324 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8c7152-824d-4b46-872c-29c108581721 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:16.184309 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1287/2581] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:03:16 2022] POST /identity/v3/projects => generated 371 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:03:16.188405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0f9d2f-f94a-4125-9183-567ae817be90 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:16.198902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0f9d2f-f94a-4125-9183-567ae817be90 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:16.216278 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:16.257719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafaf67e-d7fd-4506-8fdf-7cfe89d55867 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:16.258440 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafaf67e-d7fd-4506-8fdf-7cfe89d55867 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:16.258752 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafaf67e-d7fd-4506-8fdf-7cfe89d55867 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:16.259042 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafaf67e-d7fd-4506-8fdf-7cfe89d55867 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:16.259653 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafaf67e-d7fd-4506-8fdf-7cfe89d55867 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:16.261583 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafaf67e-d7fd-4506-8fdf-7cfe89d55867 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:16.282861 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1295/2582] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:03:16 2022] POST /identity/v3/users => generated 286 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:03:16.286983 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8c7152-824d-4b46-872c-29c108581721 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:16.297781 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8c7152-824d-4b46-872c-29c108581721 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:16.302379 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465772a7-690b-4aaf-ba84-a09094de298b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:16.303003 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465772a7-690b-4aaf-ba84-a09094de298b admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:16.303280 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465772a7-690b-4aaf-ba84-a09094de298b admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:16.303549 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465772a7-690b-4aaf-ba84-a09094de298b admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:16.304188 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465772a7-690b-4aaf-ba84-a09094de298b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:16.305775 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465772a7-690b-4aaf-ba84-a09094de298b admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:16.315946 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1288/2583] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:03:16 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:03:16.319796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafaf67e-d7fd-4506-8fdf-7cfe89d55867 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:16.329647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafaf67e-d7fd-4506-8fdf-7cfe89d55867 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:16.334558 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c680ea-fe64-4898-b891-4389bff13e4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:16.335153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c680ea-fe64-4898-b891-4389bff13e4b admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:16.335432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c680ea-fe64-4898-b891-4389bff13e4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:16.335704 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c680ea-fe64-4898-b891-4389bff13e4b admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:16.336344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c680ea-fe64-4898-b891-4389bff13e4b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:16.338525 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c680ea-fe64-4898-b891-4389bff13e4b admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:16.349171 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1296/2584] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:03:16 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:03:16.353307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465772a7-690b-4aaf-ba84-a09094de298b admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:16.363189 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465772a7-690b-4aaf-ba84-a09094de298b admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:16.367796 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365311c0-9998-4ec5-ad0d-e5c6d0a0da32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:16.368924 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365311c0-9998-4ec5-ad0d-e5c6d0a0da32 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:16.369221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365311c0-9998-4ec5-ad0d-e5c6d0a0da32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:16.369475 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365311c0-9998-4ec5-ad0d-e5c6d0a0da32 admin admin] PATH_INFO: `/v3/projects/5debd2c8413149fd9999fbe0634e42c4/users/f5f896f95aea4b949f90980a76c50e8f/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:16.385044 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365311c0-9998-4ec5-ad0d-e5c6d0a0da32 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5debd2c8413149fd9999fbe0634e42c4, user_id=f5f896f95aea4b949f90980a76c50e8f, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:16.387741 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365311c0-9998-4ec5-ad0d-e5c6d0a0da32 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:16.402339 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1289/2585] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:03:16 2022] PUT /identity/v3/projects/5debd2c8413149fd9999fbe0634e42c4/users/f5f896f95aea4b949f90980a76c50e8f/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:03:16.412625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9957688-d17c-4e7c-b3eb-b27b82a218c0 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:16.412912 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9957688-d17c-4e7c-b3eb-b27b82a218c0 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:16.413185 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9957688-d17c-4e7c-b3eb-b27b82a218c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:16.443830 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-c9957688-d17c-4e7c-b3eb-b27b82a218c0 None None] MFA Rules not processed for user `f5f896f95aea4b949f90980a76c50e8f`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:16.445080 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c9957688-d17c-4e7c-b3eb-b27b82a218c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:16.512893 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1297/2586] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:03:16 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:03:16.526983 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365311c0-9998-4ec5-ad0d-e5c6d0a0da32 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:16.529848 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-365311c0-9998-4ec5-ad0d-e5c6d0a0da32 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:16.715997 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365311c0-9998-4ec5-ad0d-e5c6d0a0da32 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:16.719555 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54993415-a19b-4c32-af83-fec0ab32335b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:16.720155 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54993415-a19b-4c32-af83-fec0ab32335b service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:16.720431 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54993415-a19b-4c32-af83-fec0ab32335b service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:16.720704 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54993415-a19b-4c32-af83-fec0ab32335b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:16.729678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54993415-a19b-4c32-af83-fec0ab32335b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:16.731143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54993415-a19b-4c32-af83-fec0ab32335b service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:16.740023 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1290/2587] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:03:16 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:03:16.764161 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9957688-d17c-4e7c-b3eb-b27b82a218c0 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:16.825817 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9957688-d17c-4e7c-b3eb-b27b82a218c0 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:16.830449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3e2a21-6384-40f3-b816-c7a00d8aff44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:16.831205 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3e2a21-6384-40f3-b816-c7a00d8aff44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:16.831802 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3e2a21-6384-40f3-b816-c7a00d8aff44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:16.832161 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3e2a21-6384-40f3-b816-c7a00d8aff44 admin admin] PATH_INFO: `/v3/users/f5f896f95aea4b949f90980a76c50e8f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:16.833366 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3e2a21-6384-40f3-b816-c7a00d8aff44 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f5f896f95aea4b949f90980a76c50e8f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:16.835463 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3e2a21-6384-40f3-b816-c7a00d8aff44 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:16.868932 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:16.886962 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8c3e2a21-6384-40f3-b816-c7a00d8aff44 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f5f896f95aea4b949f90980a76c50e8f'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:16.895100 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8c3e2a21-6384-40f3-b816-c7a00d8aff44 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f5f896f95aea4b949f90980a76c50e8f'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:16.909601 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8c3e2a21-6384-40f3-b816-c7a00d8aff44 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f5f896f95aea4b949f90980a76c50e8f'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:16.910284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8c3e2a21-6384-40f3-b816-c7a00d8aff44 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f5f896f95aea4b949f90980a76c50e8f'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:16.917113 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-8c3e2a21-6384-40f3-b816-c7a00d8aff44 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f5f896f95aea4b949f90980a76c50e8f'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:16.925303 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1298/2588] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:03:16 2022] DELETE /identity/v3/users/f5f896f95aea4b949f90980a76c50e8f => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:03:16.929939 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4986461a-f4e1-42a1-b3ec-f70845041cd7 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:16.930223 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4986461a-f4e1-42a1-b3ec-f70845041cd7 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:16.930522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4986461a-f4e1-42a1-b3ec-f70845041cd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:16.955361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-4986461a-f4e1-42a1-b3ec-f70845041cd7 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:16.957400 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-4986461a-f4e1-42a1-b3ec-f70845041cd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:17.020052 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:DELETE=6,keystone:SELECT=91,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:17.034667 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1291/2589] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:03:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:03:17.044505 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3e2a21-6384-40f3-b816-c7a00d8aff44 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:17.045903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-8c3e2a21-6384-40f3-b816-c7a00d8aff44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:17.105804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3e2a21-6384-40f3-b816-c7a00d8aff44 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:17.110315 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1efd4a-ef68-4d0d-9eef-0b212e34ed50 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:17.110961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1efd4a-ef68-4d0d-9eef-0b212e34ed50 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:17.111244 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1efd4a-ef68-4d0d-9eef-0b212e34ed50 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:17.111514 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1efd4a-ef68-4d0d-9eef-0b212e34ed50 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:17.120775 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff1efd4a-ef68-4d0d-9eef-0b212e34ed50 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:17.122207 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff1efd4a-ef68-4d0d-9eef-0b212e34ed50 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:17.131674 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1299/2590] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:03:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 01:03:17.406164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4986461a-f4e1-42a1-b3ec-f70845041cd7 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:17.408167 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-4986461a-f4e1-42a1-b3ec-f70845041cd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:17.421904 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4986461a-f4e1-42a1-b3ec-f70845041cd7 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:17.440041 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaceacd2-12e9-4d4e-b6d8-eb7f7883b6b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:17.440818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaceacd2-12e9-4d4e-b6d8-eb7f7883b6b6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:17.441137 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaceacd2-12e9-4d4e-b6d8-eb7f7883b6b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:17.441536 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaceacd2-12e9-4d4e-b6d8-eb7f7883b6b6 admin admin] PATH_INFO: `/v3/projects/5debd2c8413149fd9999fbe0634e42c4` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:17.442817 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaceacd2-12e9-4d4e-b6d8-eb7f7883b6b6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5debd2c8413149fd9999fbe0634e42c4)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:17.444864 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaceacd2-12e9-4d4e-b6d8-eb7f7883b6b6 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:17.494027 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-eaceacd2-12e9-4d4e-b6d8-eb7f7883b6b6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5debd2c8413149fd9999fbe0634e42c4', 'request_id': 'req-eaceacd2-12e9-4d4e-b6d8-eb7f7883b6b6'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:17.499548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-eaceacd2-12e9-4d4e-b6d8-eb7f7883b6b6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5debd2c8413149fd9999fbe0634e42c4', 'request_id': 'req-eaceacd2-12e9-4d4e-b6d8-eb7f7883b6b6'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:17.506857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-eaceacd2-12e9-4d4e-b6d8-eb7f7883b6b6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5debd2c8413149fd9999fbe0634e42c4', 'request_id': 'req-eaceacd2-12e9-4d4e-b6d8-eb7f7883b6b6'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:17.519611 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-eaceacd2-12e9-4d4e-b6d8-eb7f7883b6b6 admin admin] The token cache is being invalidate because project 5debd2c8413149fd9999fbe0634e42c4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:03:17.519900 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-eaceacd2-12e9-4d4e-b6d8-eb7f7883b6b6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:17.520930 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-eaceacd2-12e9-4d4e-b6d8-eb7f7883b6b6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:17.522140 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1292/2591] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:03:17 2022] DELETE /identity/v3/projects/5debd2c8413149fd9999fbe0634e42c4 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:03:17.527339 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a220e615-a9cd-4dce-a8b1-67b442121201 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:17.527804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a220e615-a9cd-4dce-a8b1-67b442121201 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:17.527999 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a220e615-a9cd-4dce-a8b1-67b442121201 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:17.547995 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-a220e615-a9cd-4dce-a8b1-67b442121201 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:17.549185 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a220e615-a9cd-4dce-a8b1-67b442121201 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:17.612541 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1300/2592] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:03:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 01:03:17.621002 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9370244a-1a18-4855-ae81-6ddc49c616ba None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:17.621528 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9370244a-1a18-4855-ae81-6ddc49c616ba None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:17.621528 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9370244a-1a18-4855-ae81-6ddc49c616ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:17.640919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-9370244a-1a18-4855-ae81-6ddc49c616ba None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:17.642146 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-9370244a-1a18-4855-ae81-6ddc49c616ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:17.650072 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1293/2593] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:03:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:03:17.654478 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a220e615-a9cd-4dce-a8b1-67b442121201 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:17.664295 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a220e615-a9cd-4dce-a8b1-67b442121201 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:17.669358 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aee605b-b033-4357-9303-360fa08ea834 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:17.669959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aee605b-b033-4357-9303-360fa08ea834 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:17.670238 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aee605b-b033-4357-9303-360fa08ea834 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:17.670536 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aee605b-b033-4357-9303-360fa08ea834 admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:17.671207 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aee605b-b033-4357-9303-360fa08ea834 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:03:17.671510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aee605b-b033-4357-9303-360fa08ea834 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:17.673100 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aee605b-b033-4357-9303-360fa08ea834 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:17.685187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1301/2594] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:03:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 01:03:17.689471 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9370244a-1a18-4855-ae81-6ddc49c616ba None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:17.699241 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9370244a-1a18-4855-ae81-6ddc49c616ba None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:17.703990 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3eaa6e-5f33-4371-ac1a-5013639196cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:17.704595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3eaa6e-5f33-4371-ac1a-5013639196cd admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:17.704868 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3eaa6e-5f33-4371-ac1a-5013639196cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:17.705135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3eaa6e-5f33-4371-ac1a-5013639196cd admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:17.705773 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3eaa6e-5f33-4371-ac1a-5013639196cd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:17.707927 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3eaa6e-5f33-4371-ac1a-5013639196cd admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:17.738878 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1294/2595] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:03:17 2022] POST /identity/v3/projects => generated 371 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:03:17.739997 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aee605b-b033-4357-9303-360fa08ea834 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:17.750853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aee605b-b033-4357-9303-360fa08ea834 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:17.785207 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:17.814643 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efe17f1-f0a7-4467-a560-5bc799a6c5e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:17.815251 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efe17f1-f0a7-4467-a560-5bc799a6c5e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:17.815528 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efe17f1-f0a7-4467-a560-5bc799a6c5e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:17.815798 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efe17f1-f0a7-4467-a560-5bc799a6c5e7 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:17.816483 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efe17f1-f0a7-4467-a560-5bc799a6c5e7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:17.819056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efe17f1-f0a7-4467-a560-5bc799a6c5e7 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:17.848181 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1302/2596] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:03:17 2022] POST /identity/v3/users => generated 286 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 01:03:17.849588 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3eaa6e-5f33-4371-ac1a-5013639196cd admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:17.860459 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3eaa6e-5f33-4371-ac1a-5013639196cd admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:17.865247 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6768c6b-5328-4008-a410-ce557fe2ac95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:17.865844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6768c6b-5328-4008-a410-ce557fe2ac95 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:17.866217 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6768c6b-5328-4008-a410-ce557fe2ac95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:17.866540 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6768c6b-5328-4008-a410-ce557fe2ac95 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:17.867184 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6768c6b-5328-4008-a410-ce557fe2ac95 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:17.868833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6768c6b-5328-4008-a410-ce557fe2ac95 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:17.882958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1295/2597] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:03:17 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:03:17.884601 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efe17f1-f0a7-4467-a560-5bc799a6c5e7 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:17.894731 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efe17f1-f0a7-4467-a560-5bc799a6c5e7 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:17.899566 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a388d2cb-66c2-406a-a802-9c2d8e6c121a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:17.901360 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a388d2cb-66c2-406a-a802-9c2d8e6c121a admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:17.901360 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a388d2cb-66c2-406a-a802-9c2d8e6c121a admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:17.901360 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a388d2cb-66c2-406a-a802-9c2d8e6c121a admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:17.901981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a388d2cb-66c2-406a-a802-9c2d8e6c121a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:17.903906 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a388d2cb-66c2-406a-a802-9c2d8e6c121a admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:17.915708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1303/2598] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:03:17 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:03:17.920545 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6768c6b-5328-4008-a410-ce557fe2ac95 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:17.930802 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6768c6b-5328-4008-a410-ce557fe2ac95 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:17.936313 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48339852-8348-467a-a310-2d23745829c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:17.936962 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48339852-8348-467a-a310-2d23745829c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:17.937245 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48339852-8348-467a-a310-2d23745829c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:17.937519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48339852-8348-467a-a310-2d23745829c2 admin admin] PATH_INFO: `/v3/projects/54761854b3e9496ea28cc4c8224c25bf/users/7d5aef87230d4424a73e87248ae81d78/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:17.954151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48339852-8348-467a-a310-2d23745829c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=54761854b3e9496ea28cc4c8224c25bf, user_id=7d5aef87230d4424a73e87248ae81d78, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:17.958159 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48339852-8348-467a-a310-2d23745829c2 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:17.973044 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1296/2599] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:03:17 2022] PUT /identity/v3/projects/54761854b3e9496ea28cc4c8224c25bf/users/7d5aef87230d4424a73e87248ae81d78/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:03:17.985285 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02195537-6cf0-4473-80d0-8e72f70a72b7 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:17.985597 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02195537-6cf0-4473-80d0-8e72f70a72b7 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:17.985877 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02195537-6cf0-4473-80d0-8e72f70a72b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:18.014957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-02195537-6cf0-4473-80d0-8e72f70a72b7 None None] MFA Rules not processed for user `7d5aef87230d4424a73e87248ae81d78`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:18.016195 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-02195537-6cf0-4473-80d0-8e72f70a72b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:18.087625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1304/2600] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:03:17 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:03:18.098213 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48339852-8348-467a-a310-2d23745829c2 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:18.099991 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-48339852-8348-467a-a310-2d23745829c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:18.157795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48339852-8348-467a-a310-2d23745829c2 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:18.161541 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c29f2a-db4b-4825-80b7-058d6532082b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:18.162140 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c29f2a-db4b-4825-80b7-058d6532082b service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:18.162415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c29f2a-db4b-4825-80b7-058d6532082b service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:18.162723 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c29f2a-db4b-4825-80b7-058d6532082b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:18.171222 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c29f2a-db4b-4825-80b7-058d6532082b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:18.172646 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c29f2a-db4b-4825-80b7-058d6532082b service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:18.182175 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1297/2601] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:03:18 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:03:19.560047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690aef4a-79a4-4f4d-a8d8-623f644de025 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:19.560551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690aef4a-79a4-4f4d-a8d8-623f644de025 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:19.560647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690aef4a-79a4-4f4d-a8d8-623f644de025 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:19.579879 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-690aef4a-79a4-4f4d-a8d8-623f644de025 None None] MFA Rules not processed for user `7d5aef87230d4424a73e87248ae81d78`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:19.581060 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-690aef4a-79a4-4f4d-a8d8-623f644de025 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:19.588502 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1305/2602] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:03:19 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:03:21.111042 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe45fb5-9679-42f4-9eae-fc3f1f1616ab None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:21.112347 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe45fb5-9679-42f4-9eae-fc3f1f1616ab None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:21.112347 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe45fb5-9679-42f4-9eae-fc3f1f1616ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:21.134102 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-3fe45fb5-9679-42f4-9eae-fc3f1f1616ab None None] MFA Rules not processed for user `3e31dce69c344c9aa4a55f31b9c735df`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:21.135411 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-3fe45fb5-9679-42f4-9eae-fc3f1f1616ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:21.145011 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:21.192779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1298/2603] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 18 01:03:21 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:03:21.206089 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690aef4a-79a4-4f4d-a8d8-623f644de025 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:21.207859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-690aef4a-79a4-4f4d-a8d8-623f644de025 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:21.300315 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690aef4a-79a4-4f4d-a8d8-623f644de025 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:21.306116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8111bf-b271-41b0-872c-a00dee8a9155 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6230ed7ce0eb43e9b2b8530a28eb6299', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:21.306752 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8111bf-b271-41b0-872c-a00dee8a9155 service swift] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:21.307032 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8111bf-b271-41b0-872c-a00dee8a9155 service swift] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:21.307303 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8111bf-b271-41b0-872c-a00dee8a9155 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:21.318325 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8111bf-b271-41b0-872c-a00dee8a9155 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:21.319975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8111bf-b271-41b0-872c-a00dee8a9155 service swift] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:21.328786 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1306/2604] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 18 01:03:21 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 18 01:03:31.186073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=18 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:31.325369 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=84,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:35.634329 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42aa06e8-4ef9-467b-89fd-e064ac1da1bb None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:35.635278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42aa06e8-4ef9-467b-89fd-e064ac1da1bb None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:35.636685 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42aa06e8-4ef9-467b-89fd-e064ac1da1bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:35.660999 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-42aa06e8-4ef9-467b-89fd-e064ac1da1bb None None] MFA Rules not processed for user `3e31dce69c344c9aa4a55f31b9c735df`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:35.662510 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-42aa06e8-4ef9-467b-89fd-e064ac1da1bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:35.675900 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1299/2605] 10.222.0.42 () {58 vars in 967 bytes} [Sun Sep 18 01:03:35 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:03:35.684503 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8111bf-b271-41b0-872c-a00dee8a9155 service swift] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:35.695862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8111bf-b271-41b0-872c-a00dee8a9155 service swift] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:35.701709 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b516bd69-7421-4f44-80bd-e125a47f9db3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6230ed7ce0eb43e9b2b8530a28eb6299', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:35.702405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b516bd69-7421-4f44-80bd-e125a47f9db3 service swift] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:35.702802 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b516bd69-7421-4f44-80bd-e125a47f9db3 service swift] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:35.703158 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b516bd69-7421-4f44-80bd-e125a47f9db3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:35.712224 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b516bd69-7421-4f44-80bd-e125a47f9db3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:35.714525 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b516bd69-7421-4f44-80bd-e125a47f9db3 service swift] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:35.723194 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1307/2606] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 18 01:03:35 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 18 01:03:37.436508 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa713ec9-0617-4ea5-890e-b76d4a4bf94d None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:37.437003 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa713ec9-0617-4ea5-890e-b76d4a4bf94d None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:37.437110 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa713ec9-0617-4ea5-890e-b76d4a4bf94d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:37.458711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-aa713ec9-0617-4ea5-890e-b76d4a4bf94d None None] MFA Rules not processed for user `823fa0c3eeeb489591c639b81afc02bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:37.459938 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-aa713ec9-0617-4ea5-890e-b76d4a4bf94d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:37.516123 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1300/2607] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:03:37 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 01:03:37.846314 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b516bd69-7421-4f44-80bd-e125a47f9db3 service swift] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:37.847744 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-b516bd69-7421-4f44-80bd-e125a47f9db3 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:37.905031 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b516bd69-7421-4f44-80bd-e125a47f9db3 service swift] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:37.908878 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa15515-ba4a-4870-a0ed-8a107f178f8d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:37.909498 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa15515-ba4a-4870-a0ed-8a107f178f8d service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:37.909870 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa15515-ba4a-4870-a0ed-8a107f178f8d service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:37.910238 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa15515-ba4a-4870-a0ed-8a107f178f8d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:37.912053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3aa15515-ba4a-4870-a0ed-8a107f178f8d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:37.922504 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa15515-ba4a-4870-a0ed-8a107f178f8d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:37.923947 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa15515-ba4a-4870-a0ed-8a107f178f8d service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:37.932125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1308/2608] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:03:37 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:03:47.508578 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=28 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:47.928384 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=32 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:50.267972 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522cd2e6-7bc7-4445-a3ae-bb10a294be4d None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:50.268516 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522cd2e6-7bc7-4445-a3ae-bb10a294be4d None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:50.268628 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522cd2e6-7bc7-4445-a3ae-bb10a294be4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:50.296019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-522cd2e6-7bc7-4445-a3ae-bb10a294be4d None None] MFA Rules not processed for user `3e31dce69c344c9aa4a55f31b9c735df`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:50.297506 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-522cd2e6-7bc7-4445-a3ae-bb10a294be4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:50.305732 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1301/2609] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 18 01:03:50 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:03:50.318066 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa15515-ba4a-4870-a0ed-8a107f178f8d service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:50.331193 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa15515-ba4a-4870-a0ed-8a107f178f8d service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:50.336544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc08181-0c27-4529-a4ae-2361712eb873 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6230ed7ce0eb43e9b2b8530a28eb6299', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:50.337153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc08181-0c27-4529-a4ae-2361712eb873 service swift] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:50.337433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc08181-0c27-4529-a4ae-2361712eb873 service swift] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:50.337708 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc08181-0c27-4529-a4ae-2361712eb873 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:50.347281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc08181-0c27-4529-a4ae-2361712eb873 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:50.349153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc08181-0c27-4529-a4ae-2361712eb873 service swift] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:50.358951 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1309/2610] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 18 01:03:50 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 18 01:03:50.919482 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d642ce39-58be-470b-a16e-c6dffafcc223 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:50.919767 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d642ce39-58be-470b-a16e-c6dffafcc223 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:50.920039 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d642ce39-58be-470b-a16e-c6dffafcc223 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:50.940223 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-d642ce39-58be-470b-a16e-c6dffafcc223 None None] MFA Rules not processed for user `3e31dce69c344c9aa4a55f31b9c735df`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:50.941417 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d642ce39-58be-470b-a16e-c6dffafcc223 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:50.950333 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1302/2611] 10.222.0.42 () {58 vars in 967 bytes} [Sun Sep 18 01:03:50 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:03:50.963163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc08181-0c27-4529-a4ae-2361712eb873 service swift] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:50.973781 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc08181-0c27-4529-a4ae-2361712eb873 service swift] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:50.978891 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ef3815-bfb5-4c5e-9746-e40f95af45c0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6230ed7ce0eb43e9b2b8530a28eb6299', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:50.979594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ef3815-bfb5-4c5e-9746-e40f95af45c0 service swift] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:50.980053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ef3815-bfb5-4c5e-9746-e40f95af45c0 service swift] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:50.980396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ef3815-bfb5-4c5e-9746-e40f95af45c0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:50.988758 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ef3815-bfb5-4c5e-9746-e40f95af45c0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:50.990514 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ef3815-bfb5-4c5e-9746-e40f95af45c0 service swift] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:50.999743 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1310/2612] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 18 01:03:50 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 18 01:03:51.238005 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b774e5-047b-4e04-9a48-629342d80126 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:51.238280 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b774e5-047b-4e04-9a48-629342d80126 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:51.238663 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b774e5-047b-4e04-9a48-629342d80126 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:51.260133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-78b774e5-047b-4e04-9a48-629342d80126 None None] MFA Rules not processed for user `3e31dce69c344c9aa4a55f31b9c735df`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:51.262196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-78b774e5-047b-4e04-9a48-629342d80126 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:51.272134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1303/2613] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 18 01:03:51 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:03:51.285575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ef3815-bfb5-4c5e-9746-e40f95af45c0 service swift] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:51.297119 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ef3815-bfb5-4c5e-9746-e40f95af45c0 service swift] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:51.302182 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330302e7-1208-4fdb-a18d-685c377ce27d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6230ed7ce0eb43e9b2b8530a28eb6299', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:51.302839 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330302e7-1208-4fdb-a18d-685c377ce27d service swift] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:51.303123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330302e7-1208-4fdb-a18d-685c377ce27d service swift] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:51.303398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330302e7-1208-4fdb-a18d-685c377ce27d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:51.312748 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330302e7-1208-4fdb-a18d-685c377ce27d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:51.314336 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330302e7-1208-4fdb-a18d-685c377ce27d service swift] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:51.322369 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1311/2614] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 18 01:03:51 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 18 01:03:51.666171 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad000fe8-258f-4374-b271-2020faa1731b None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:51.666480 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad000fe8-258f-4374-b271-2020faa1731b None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:51.667388 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad000fe8-258f-4374-b271-2020faa1731b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:51.686701 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-ad000fe8-258f-4374-b271-2020faa1731b None None] MFA Rules not processed for user `3e31dce69c344c9aa4a55f31b9c735df`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:51.688409 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ad000fe8-258f-4374-b271-2020faa1731b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:51.695563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1304/2615] 10.222.0.42 () {58 vars in 967 bytes} [Sun Sep 18 01:03:51 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:03:51.708275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330302e7-1208-4fdb-a18d-685c377ce27d service swift] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:51.719692 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330302e7-1208-4fdb-a18d-685c377ce27d service swift] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:51.724696 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc823750-4607-4b56-a693-563da8c7c401 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6230ed7ce0eb43e9b2b8530a28eb6299', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:51.725295 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc823750-4607-4b56-a693-563da8c7c401 service swift] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:51.725609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc823750-4607-4b56-a693-563da8c7c401 service swift] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:51.725850 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc823750-4607-4b56-a693-563da8c7c401 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:51.735398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc823750-4607-4b56-a693-563da8c7c401 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:51.736965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc823750-4607-4b56-a693-563da8c7c401 service swift] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:51.745500 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1312/2616] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 18 01:03:51 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 18 01:03:53.955277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad000fe8-258f-4374-b271-2020faa1731b None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:54.048661 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad000fe8-258f-4374-b271-2020faa1731b None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:54.054182 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50d2358-1059-40e2-8b32-5251a58dc767 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:54.055222 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50d2358-1059-40e2-8b32-5251a58dc767 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:54.055363 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50d2358-1059-40e2-8b32-5251a58dc767 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:54.055591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50d2358-1059-40e2-8b32-5251a58dc767 admin admin] PATH_INFO: `/v3/users/7d5aef87230d4424a73e87248ae81d78` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:54.056728 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50d2358-1059-40e2-8b32-5251a58dc767 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7d5aef87230d4424a73e87248ae81d78)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:54.058725 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50d2358-1059-40e2-8b32-5251a58dc767 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:54.109983 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c50d2358-1059-40e2-8b32-5251a58dc767 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7d5aef87230d4424a73e87248ae81d78'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:54.126839 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c50d2358-1059-40e2-8b32-5251a58dc767 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7d5aef87230d4424a73e87248ae81d78'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:54.128143 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c50d2358-1059-40e2-8b32-5251a58dc767 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7d5aef87230d4424a73e87248ae81d78'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:54.128899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c50d2358-1059-40e2-8b32-5251a58dc767 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7d5aef87230d4424a73e87248ae81d78'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:54.138894 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c50d2358-1059-40e2-8b32-5251a58dc767 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7d5aef87230d4424a73e87248ae81d78'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:54.157032 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1305/2617] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:03:53 2022] DELETE /identity/v3/users/7d5aef87230d4424a73e87248ae81d78 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:03:54.161549 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6ab9f8-16d2-4178-9817-da84fcba8023 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:54.162004 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6ab9f8-16d2-4178-9817-da84fcba8023 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:54.162100 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6ab9f8-16d2-4178-9817-da84fcba8023 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:54.181656 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-3c6ab9f8-16d2-4178-9817-da84fcba8023 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:03:54.182917 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3c6ab9f8-16d2-4178-9817-da84fcba8023 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:54.245029 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1313/2618] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:03:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 01:03:54.254947 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50d2358-1059-40e2-8b32-5251a58dc767 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:54.256462 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c50d2358-1059-40e2-8b32-5251a58dc767 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:54.322562 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50d2358-1059-40e2-8b32-5251a58dc767 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:54.326880 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe11f87-176c-40a7-82cc-97b79b9f1912 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:54.327498 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe11f87-176c-40a7-82cc-97b79b9f1912 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:54.327776 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe11f87-176c-40a7-82cc-97b79b9f1912 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:54.328072 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe11f87-176c-40a7-82cc-97b79b9f1912 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:54.336503 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:03:54.339946 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe11f87-176c-40a7-82cc-97b79b9f1912 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:54.342514 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe11f87-176c-40a7-82cc-97b79b9f1912 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:54.353286 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1306/2619] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:03:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:03:54.537332 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6ab9f8-16d2-4178-9817-da84fcba8023 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:54.539821 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3c6ab9f8-16d2-4178-9817-da84fcba8023 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:54.553057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6ab9f8-16d2-4178-9817-da84fcba8023 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:54.559976 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc168ad-75f9-4ad6-9436-07d563cd2a2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:54.561437 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc168ad-75f9-4ad6-9436-07d563cd2a2a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:54.561906 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc168ad-75f9-4ad6-9436-07d563cd2a2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:54.562380 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc168ad-75f9-4ad6-9436-07d563cd2a2a admin admin] PATH_INFO: `/v3/projects/54761854b3e9496ea28cc4c8224c25bf` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:54.563764 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc168ad-75f9-4ad6-9436-07d563cd2a2a admin admin] RBAC: Authorizing `identity:delete_project(project_id=54761854b3e9496ea28cc4c8224c25bf)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:54.566278 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc168ad-75f9-4ad6-9436-07d563cd2a2a admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:54.623841 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-0dc168ad-75f9-4ad6-9436-07d563cd2a2a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '54761854b3e9496ea28cc4c8224c25bf', 'request_id': 'req-0dc168ad-75f9-4ad6-9436-07d563cd2a2a'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:54.630272 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-0dc168ad-75f9-4ad6-9436-07d563cd2a2a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '54761854b3e9496ea28cc4c8224c25bf', 'request_id': 'req-0dc168ad-75f9-4ad6-9436-07d563cd2a2a'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:54.638925 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-0dc168ad-75f9-4ad6-9436-07d563cd2a2a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '54761854b3e9496ea28cc4c8224c25bf', 'request_id': 'req-0dc168ad-75f9-4ad6-9436-07d563cd2a2a'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:54.650269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-0dc168ad-75f9-4ad6-9436-07d563cd2a2a admin admin] The token cache is being invalidate because project 54761854b3e9496ea28cc4c8224c25bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:03:54.650590 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-0dc168ad-75f9-4ad6-9436-07d563cd2a2a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:54.652002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-0dc168ad-75f9-4ad6-9436-07d563cd2a2a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:03:54.653482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1314/2620] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:03:54 2022] DELETE /identity/v3/projects/54761854b3e9496ea28cc4c8224c25bf => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:03:56.508010 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe11f87-176c-40a7-82cc-97b79b9f1912 service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:03:56.518581 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe11f87-176c-40a7-82cc-97b79b9f1912 service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:03:56.575123 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d64f542-0eec-4be1-add7-525a9b81b0f6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:03:56.575770 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d64f542-0eec-4be1-add7-525a9b81b0f6 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:03:56.576044 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d64f542-0eec-4be1-add7-525a9b81b0f6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:03:56.576324 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d64f542-0eec-4be1-add7-525a9b81b0f6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:03:56.578161 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-0d64f542-0eec-4be1-add7-525a9b81b0f6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:03:56.646140 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d64f542-0eec-4be1-add7-525a9b81b0f6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:03:56.647635 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d64f542-0eec-4be1-add7-525a9b81b0f6 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:03:56.658470 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1307/2621] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:03:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:04:00.027652 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70454ca9-15c8-41ad-b7cb-6bceb791f44c None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:00.028146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70454ca9-15c8-41ad-b7cb-6bceb791f44c None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:00.028259 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70454ca9-15c8-41ad-b7cb-6bceb791f44c None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:00.029541 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1315/2622] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 18 01:04:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 01:04:00.038778 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848347b9-ebb2-4ee6-874d-fdca66e5c1f5 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:00.039232 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848347b9-ebb2-4ee6-874d-fdca66e5c1f5 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:00.039380 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848347b9-ebb2-4ee6-874d-fdca66e5c1f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:00.060459 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-848347b9-ebb2-4ee6-874d-fdca66e5c1f5 None None] MFA Rules not processed for user `8f30116fb5ce43dab69efa8bafd9d049`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:00.061679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-848347b9-ebb2-4ee6-874d-fdca66e5c1f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:00.117791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70454ca9-15c8-41ad-b7cb-6bceb791f44c None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:00.119628 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-70454ca9-15c8-41ad-b7cb-6bceb791f44c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:00.136146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70454ca9-15c8-41ad-b7cb-6bceb791f44c None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:00.139232 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1308/2623] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 18 01:04:00 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:00.141474 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f2855d-547d-4a14-b797-f58958b6fa6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:00.142200 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f2855d-547d-4a14-b797-f58958b6fa6a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:00.142571 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f2855d-547d-4a14-b797-f58958b6fa6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:00.142912 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f2855d-547d-4a14-b797-f58958b6fa6a admin admin] PATH_INFO: `/v3/users/2f9751adf8d74ab59b6b0a5d00221405` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:00.144205 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f2855d-547d-4a14-b797-f58958b6fa6a admin admin] RBAC: Authorizing `identity:delete_user(user_id=2f9751adf8d74ab59b6b0a5d00221405)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:00.146250 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f2855d-547d-4a14-b797-f58958b6fa6a admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:00.149598 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848347b9-ebb2-4ee6-874d-fdca66e5c1f5 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:00.151053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-848347b9-ebb2-4ee6-874d-fdca66e5c1f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:00.164026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848347b9-ebb2-4ee6-874d-fdca66e5c1f5 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:00.169618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15cc50e8-adca-49b2-a857-482d2103dc0a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8f30116fb5ce43dab69efa8bafd9d049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:00.170223 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15cc50e8-adca-49b2-a857-482d2103dc0a service placement] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:00.170525 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15cc50e8-adca-49b2-a857-482d2103dc0a service placement] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:00.170805 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15cc50e8-adca-49b2-a857-482d2103dc0a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:00.176207 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:00.180725 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15cc50e8-adca-49b2-a857-482d2103dc0a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:00.182445 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15cc50e8-adca-49b2-a857-482d2103dc0a service placement] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:00.194632 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1309/2624] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:04:00 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 01:04:00.199935 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a4f2855d-547d-4a14-b797-f58958b6fa6a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2f9751adf8d74ab59b6b0a5d00221405'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:00.209342 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a4f2855d-547d-4a14-b797-f58958b6fa6a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2f9751adf8d74ab59b6b0a5d00221405'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:00.222179 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a4f2855d-547d-4a14-b797-f58958b6fa6a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2f9751adf8d74ab59b6b0a5d00221405'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:00.222910 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a4f2855d-547d-4a14-b797-f58958b6fa6a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2f9751adf8d74ab59b6b0a5d00221405'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:00.230637 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a4f2855d-547d-4a14-b797-f58958b6fa6a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2f9751adf8d74ab59b6b0a5d00221405'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:00.236664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1316/2625] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:04:00 2022] DELETE /identity/v3/users/2f9751adf8d74ab59b6b0a5d00221405 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:00.429575 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15cc50e8-adca-49b2-a857-482d2103dc0a service placement] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:00.442367 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15cc50e8-adca-49b2-a857-482d2103dc0a service placement] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:00.470273 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:00.504117 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5f9ca9-5747-4026-94de-46af99037046 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:00.504801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5f9ca9-5747-4026-94de-46af99037046 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:00.505119 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5f9ca9-5747-4026-94de-46af99037046 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:00.505433 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5f9ca9-5747-4026-94de-46af99037046 admin admin] PATH_INFO: `/v3/projects/cf1375f9f28c43fda090a236635c4188` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:00.506661 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5f9ca9-5747-4026-94de-46af99037046 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cf1375f9f28c43fda090a236635c4188)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:00.513588 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5f9ca9-5747-4026-94de-46af99037046 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:00.561226 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0a5f9ca9-5747-4026-94de-46af99037046 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cf1375f9f28c43fda090a236635c4188', 'request_id': 'req-0a5f9ca9-5747-4026-94de-46af99037046'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:00.566638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0a5f9ca9-5747-4026-94de-46af99037046 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cf1375f9f28c43fda090a236635c4188', 'request_id': 'req-0a5f9ca9-5747-4026-94de-46af99037046'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:00.573978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0a5f9ca9-5747-4026-94de-46af99037046 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cf1375f9f28c43fda090a236635c4188', 'request_id': 'req-0a5f9ca9-5747-4026-94de-46af99037046'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:00.585370 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0a5f9ca9-5747-4026-94de-46af99037046 admin admin] The token cache is being invalidate because project cf1375f9f28c43fda090a236635c4188 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:04:00.585654 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0a5f9ca9-5747-4026-94de-46af99037046 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:00.586284 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0a5f9ca9-5747-4026-94de-46af99037046 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:00.587774 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1310/2626] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:04:00 2022] DELETE /identity/v3/projects/cf1375f9f28c43fda090a236635c4188 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:00.592691 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0b5b42-4b04-44bb-9f46-ce43cf1fd474 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:00.593053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0b5b42-4b04-44bb-9f46-ce43cf1fd474 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:00.593397 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0b5b42-4b04-44bb-9f46-ce43cf1fd474 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:00.615131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-4d0b5b42-4b04-44bb-9f46-ce43cf1fd474 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:00.616922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-4d0b5b42-4b04-44bb-9f46-ce43cf1fd474 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:00.685101 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1317/2627] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:00.693058 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce25fdb-1c2e-403b-8112-11e8f725b0b9 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:00.693341 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce25fdb-1c2e-403b-8112-11e8f725b0b9 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:00.693639 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce25fdb-1c2e-403b-8112-11e8f725b0b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:00.713518 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-3ce25fdb-1c2e-403b-8112-11e8f725b0b9 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:00.715304 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-3ce25fdb-1c2e-403b-8112-11e8f725b0b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:00.723305 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1311/2628] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:00.727484 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0b5b42-4b04-44bb-9f46-ce43cf1fd474 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:00.737788 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0b5b42-4b04-44bb-9f46-ce43cf1fd474 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:00.742379 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d624c9-d197-44a5-99b2-3660db64d1dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:00.743078 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d624c9-d197-44a5-99b2-3660db64d1dc admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:00.743333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d624c9-d197-44a5-99b2-3660db64d1dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:00.743609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d624c9-d197-44a5-99b2-3660db64d1dc admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:00.744319 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d624c9-d197-44a5-99b2-3660db64d1dc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:04:00.744647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d624c9-d197-44a5-99b2-3660db64d1dc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:00.746491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d624c9-d197-44a5-99b2-3660db64d1dc admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:00.759173 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1318/2629] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:04:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 01:04:00.763511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce25fdb-1c2e-403b-8112-11e8f725b0b9 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:00.772582 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce25fdb-1c2e-403b-8112-11e8f725b0b9 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:00.777794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f0c2d4-8e61-4ee2-8f86-22ff17e32265 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:00.778379 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f0c2d4-8e61-4ee2-8f86-22ff17e32265 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:00.778682 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f0c2d4-8e61-4ee2-8f86-22ff17e32265 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:00.779041 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f0c2d4-8e61-4ee2-8f86-22ff17e32265 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:00.779552 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f0c2d4-8e61-4ee2-8f86-22ff17e32265 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:00.781413 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f0c2d4-8e61-4ee2-8f86-22ff17e32265 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:00.809609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1312/2630] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:04:00 2022] POST /identity/v3/projects => generated 385 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 01:04:00.813923 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d624c9-d197-44a5-99b2-3660db64d1dc admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:00.824768 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d624c9-d197-44a5-99b2-3660db64d1dc admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:00.882077 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461761a6-fe60-4e92-aa79-23fe972b3e57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:00.882713 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461761a6-fe60-4e92-aa79-23fe972b3e57 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:00.882993 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461761a6-fe60-4e92-aa79-23fe972b3e57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:00.883263 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461761a6-fe60-4e92-aa79-23fe972b3e57 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:00.883899 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461761a6-fe60-4e92-aa79-23fe972b3e57 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:00.885796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461761a6-fe60-4e92-aa79-23fe972b3e57 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:00.907316 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1319/2631] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:04:00 2022] POST /identity/v3/users => generated 293 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:04:00.911396 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f0c2d4-8e61-4ee2-8f86-22ff17e32265 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:00.922319 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f0c2d4-8e61-4ee2-8f86-22ff17e32265 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:00.927037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f8d4b0-b789-442e-a7bb-b3c0e6384bd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:00.927627 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f8d4b0-b789-442e-a7bb-b3c0e6384bd2 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:00.927900 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f8d4b0-b789-442e-a7bb-b3c0e6384bd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:00.928280 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f8d4b0-b789-442e-a7bb-b3c0e6384bd2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:00.928941 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f8d4b0-b789-442e-a7bb-b3c0e6384bd2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:00.930527 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f8d4b0-b789-442e-a7bb-b3c0e6384bd2 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:00.941734 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1313/2632] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:04:00 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:04:00.945692 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461761a6-fe60-4e92-aa79-23fe972b3e57 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:00.955483 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461761a6-fe60-4e92-aa79-23fe972b3e57 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:00.960328 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10a5d3d-0a45-4dca-bf7c-a27c7109192b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:00.960917 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10a5d3d-0a45-4dca-bf7c-a27c7109192b admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:00.961192 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10a5d3d-0a45-4dca-bf7c-a27c7109192b admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:00.961461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10a5d3d-0a45-4dca-bf7c-a27c7109192b admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:00.962093 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10a5d3d-0a45-4dca-bf7c-a27c7109192b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:00.963719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10a5d3d-0a45-4dca-bf7c-a27c7109192b admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:00.974629 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1320/2633] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:04:00 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:04:00.978637 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f8d4b0-b789-442e-a7bb-b3c0e6384bd2 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:00.988978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f8d4b0-b789-442e-a7bb-b3c0e6384bd2 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:00.993569 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ca074f-ec40-4a3a-8d00-4537b6b4421f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:00.994201 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ca074f-ec40-4a3a-8d00-4537b6b4421f admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:00.994505 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ca074f-ec40-4a3a-8d00-4537b6b4421f admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:00.994782 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ca074f-ec40-4a3a-8d00-4537b6b4421f admin admin] PATH_INFO: `/v3/projects/7806d126c00c41dabdc84f853f13becc/users/d19b2f92e46148c5a3d53c3ddffede77/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:01.009527 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ca074f-ec40-4a3a-8d00-4537b6b4421f admin admin] RBAC: Authorizing `identity:create_grant(project_id=7806d126c00c41dabdc84f853f13becc, user_id=d19b2f92e46148c5a3d53c3ddffede77, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:01.012911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ca074f-ec40-4a3a-8d00-4537b6b4421f admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:01.026103 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1314/2634] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:04:00 2022] PUT /identity/v3/projects/7806d126c00c41dabdc84f853f13becc/users/d19b2f92e46148c5a3d53c3ddffede77/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:01.036109 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc77d1af-4bcb-4083-8230-9564b28e67ed None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:01.036617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc77d1af-4bcb-4083-8230-9564b28e67ed None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:01.036729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc77d1af-4bcb-4083-8230-9564b28e67ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:01.059460 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:01.072664 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-dc77d1af-4bcb-4083-8230-9564b28e67ed None None] MFA Rules not processed for user `d19b2f92e46148c5a3d53c3ddffede77`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:01.077276 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-dc77d1af-4bcb-4083-8230-9564b28e67ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:01.145659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1321/2635] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:01 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:01.157779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ca074f-ec40-4a3a-8d00-4537b6b4421f admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:01.159665 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-14ca074f-ec40-4a3a-8d00-4537b6b4421f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:01.215176 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:01.236353 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ca074f-ec40-4a3a-8d00-4537b6b4421f admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:01.241933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c75699-eced-4f98-b4e0-3c421666daab service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:01.242575 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c75699-eced-4f98-b4e0-3c421666daab service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:01.242855 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c75699-eced-4f98-b4e0-3c421666daab service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:01.243127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c75699-eced-4f98-b4e0-3c421666daab service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:01.251769 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c75699-eced-4f98-b4e0-3c421666daab service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:01.253337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c75699-eced-4f98-b4e0-3c421666daab service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:01.262570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1315/2636] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:04:01 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:04:08.560804 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc77d1af-4bcb-4083-8230-9564b28e67ed None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:08.630444 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc77d1af-4bcb-4083-8230-9564b28e67ed None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:08.634942 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cafeaf-d56f-4fe9-ba12-a5e49c5944b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:08.635567 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cafeaf-d56f-4fe9-ba12-a5e49c5944b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:08.635844 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cafeaf-d56f-4fe9-ba12-a5e49c5944b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:08.636125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cafeaf-d56f-4fe9-ba12-a5e49c5944b3 admin admin] PATH_INFO: `/v3/users/d19b2f92e46148c5a3d53c3ddffede77` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:08.637232 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cafeaf-d56f-4fe9-ba12-a5e49c5944b3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d19b2f92e46148c5a3d53c3ddffede77)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:08.639271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cafeaf-d56f-4fe9-ba12-a5e49c5944b3 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:08.686249 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-21cafeaf-d56f-4fe9-ba12-a5e49c5944b3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd19b2f92e46148c5a3d53c3ddffede77'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:08.694621 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-21cafeaf-d56f-4fe9-ba12-a5e49c5944b3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd19b2f92e46148c5a3d53c3ddffede77'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:08.707185 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-21cafeaf-d56f-4fe9-ba12-a5e49c5944b3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd19b2f92e46148c5a3d53c3ddffede77'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:08.707835 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-21cafeaf-d56f-4fe9-ba12-a5e49c5944b3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd19b2f92e46148c5a3d53c3ddffede77'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:08.715087 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-21cafeaf-d56f-4fe9-ba12-a5e49c5944b3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd19b2f92e46148c5a3d53c3ddffede77'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:08.720865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1322/2637] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:04:08 2022] DELETE /identity/v3/users/d19b2f92e46148c5a3d53c3ddffede77 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:08.725343 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f31078-a42d-409b-baa8-d58237cd3c18 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:08.725820 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f31078-a42d-409b-baa8-d58237cd3c18 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:08.725919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f31078-a42d-409b-baa8-d58237cd3c18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:08.745027 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-b8f31078-a42d-409b-baa8-d58237cd3c18 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:08.746839 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-b8f31078-a42d-409b-baa8-d58237cd3c18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:08.802262 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1316/2638] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:08.812738 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cafeaf-d56f-4fe9-ba12-a5e49c5944b3 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:08.814243 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-21cafeaf-d56f-4fe9-ba12-a5e49c5944b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:08.835128 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:08.876501 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cafeaf-d56f-4fe9-ba12-a5e49c5944b3 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:08.880136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd1b238-c236-4b9a-a33a-f5719783a42d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:08.880874 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd1b238-c236-4b9a-a33a-f5719783a42d service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:08.881150 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd1b238-c236-4b9a-a33a-f5719783a42d service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:08.881488 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd1b238-c236-4b9a-a33a-f5719783a42d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:08.890288 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd1b238-c236-4b9a-a33a-f5719783a42d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:08.891866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd1b238-c236-4b9a-a33a-f5719783a42d service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:08.901075 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1323/2639] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:04:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:04:09.162522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f31078-a42d-409b-baa8-d58237cd3c18 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:09.163982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-b8f31078-a42d-409b-baa8-d58237cd3c18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:09.178692 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f31078-a42d-409b-baa8-d58237cd3c18 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:09.183768 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ac8001-7636-4725-9eb8-9a9e5e95da4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:09.184417 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ac8001-7636-4725-9eb8-9a9e5e95da4e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:09.184698 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ac8001-7636-4725-9eb8-9a9e5e95da4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:09.185525 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ac8001-7636-4725-9eb8-9a9e5e95da4e admin admin] PATH_INFO: `/v3/projects/7806d126c00c41dabdc84f853f13becc` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:09.186877 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ac8001-7636-4725-9eb8-9a9e5e95da4e admin admin] RBAC: Authorizing `identity:delete_project(project_id=7806d126c00c41dabdc84f853f13becc)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:09.188946 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ac8001-7636-4725-9eb8-9a9e5e95da4e admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:09.238332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-16ac8001-7636-4725-9eb8-9a9e5e95da4e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7806d126c00c41dabdc84f853f13becc', 'request_id': 'req-16ac8001-7636-4725-9eb8-9a9e5e95da4e'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:09.243840 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-16ac8001-7636-4725-9eb8-9a9e5e95da4e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7806d126c00c41dabdc84f853f13becc', 'request_id': 'req-16ac8001-7636-4725-9eb8-9a9e5e95da4e'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:09.250912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-16ac8001-7636-4725-9eb8-9a9e5e95da4e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7806d126c00c41dabdc84f853f13becc', 'request_id': 'req-16ac8001-7636-4725-9eb8-9a9e5e95da4e'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:09.262650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-16ac8001-7636-4725-9eb8-9a9e5e95da4e admin admin] The token cache is being invalidate because project 7806d126c00c41dabdc84f853f13becc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:04:09.262940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-16ac8001-7636-4725-9eb8-9a9e5e95da4e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:09.263585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-16ac8001-7636-4725-9eb8-9a9e5e95da4e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:09.265104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1317/2640] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:04:09 2022] DELETE /identity/v3/projects/7806d126c00c41dabdc84f853f13becc => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:09.270035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffbe80b-8706-45e7-93db-d1175a780594 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:09.270319 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffbe80b-8706-45e7-93db-d1175a780594 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:09.270628 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffbe80b-8706-45e7-93db-d1175a780594 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:09.290630 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-8ffbe80b-8706-45e7-93db-d1175a780594 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:09.291844 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-8ffbe80b-8706-45e7-93db-d1175a780594 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:09.352314 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1324/2641] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 01:04:09.363227 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93575b8-50ea-4a73-89f3-5c501e7663a1 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:09.363512 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93575b8-50ea-4a73-89f3-5c501e7663a1 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:09.363789 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93575b8-50ea-4a73-89f3-5c501e7663a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:09.383299 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-a93575b8-50ea-4a73-89f3-5c501e7663a1 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:09.384483 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-a93575b8-50ea-4a73-89f3-5c501e7663a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:09.392927 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1318/2642] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:09.397238 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffbe80b-8706-45e7-93db-d1175a780594 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:09.407687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffbe80b-8706-45e7-93db-d1175a780594 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:09.412175 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bef798-ef58-47cf-b0cf-65d300275782 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:09.412766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bef798-ef58-47cf-b0cf-65d300275782 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:09.413043 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bef798-ef58-47cf-b0cf-65d300275782 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:09.413313 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bef798-ef58-47cf-b0cf-65d300275782 admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:09.413977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2bef798-ef58-47cf-b0cf-65d300275782 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:04:09.414276 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2bef798-ef58-47cf-b0cf-65d300275782 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:09.415901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2bef798-ef58-47cf-b0cf-65d300275782 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:09.427279 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1325/2643] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:04:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 01:04:09.431513 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93575b8-50ea-4a73-89f3-5c501e7663a1 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:09.440841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93575b8-50ea-4a73-89f3-5c501e7663a1 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:09.445645 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ef3938-66bb-4846-89ea-561480d2a0de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:09.446243 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ef3938-66bb-4846-89ea-561480d2a0de admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:09.446556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ef3938-66bb-4846-89ea-561480d2a0de admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:09.446830 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ef3938-66bb-4846-89ea-561480d2a0de admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:09.447468 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ef3938-66bb-4846-89ea-561480d2a0de admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:09.449907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ef3938-66bb-4846-89ea-561480d2a0de admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:09.479126 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1319/2644] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:04:09 2022] POST /identity/v3/projects => generated 401 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:04:09.483619 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bef798-ef58-47cf-b0cf-65d300275782 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:09.493536 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bef798-ef58-47cf-b0cf-65d300275782 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:09.552590 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41cf1bde-01f8-4cb7-a624-309be051a9eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:09.553283 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cf1bde-01f8-4cb7-a624-309be051a9eb admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:09.553566 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cf1bde-01f8-4cb7-a624-309be051a9eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:09.553838 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cf1bde-01f8-4cb7-a624-309be051a9eb admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:09.554494 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41cf1bde-01f8-4cb7-a624-309be051a9eb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:09.556422 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41cf1bde-01f8-4cb7-a624-309be051a9eb admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:09.581367 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1326/2645] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:04:09 2022] POST /identity/v3/users => generated 301 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:04:09.582567 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ef3938-66bb-4846-89ea-561480d2a0de admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:09.593605 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ef3938-66bb-4846-89ea-561480d2a0de admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:09.598185 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4c15ce-41a8-40af-9705-9615cb16ec3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:09.598812 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4c15ce-41a8-40af-9705-9615cb16ec3c admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:09.599090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4c15ce-41a8-40af-9705-9615cb16ec3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:09.599358 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4c15ce-41a8-40af-9705-9615cb16ec3c admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:09.600007 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4c15ce-41a8-40af-9705-9615cb16ec3c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:09.601592 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4c15ce-41a8-40af-9705-9615cb16ec3c admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:09.611638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1320/2646] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:04:09 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:04:09.615616 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41cf1bde-01f8-4cb7-a624-309be051a9eb admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:09.626116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41cf1bde-01f8-4cb7-a624-309be051a9eb admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:09.630719 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f2243e-b3ba-4be1-af37-a31f83056598 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:09.631312 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f2243e-b3ba-4be1-af37-a31f83056598 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:09.631589 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f2243e-b3ba-4be1-af37-a31f83056598 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:09.631857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f2243e-b3ba-4be1-af37-a31f83056598 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:09.632490 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f2243e-b3ba-4be1-af37-a31f83056598 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:09.634076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f2243e-b3ba-4be1-af37-a31f83056598 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:09.644973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1327/2647] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:04:09 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:04:09.649141 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4c15ce-41a8-40af-9705-9615cb16ec3c admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:09.659905 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4c15ce-41a8-40af-9705-9615cb16ec3c admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:09.664512 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f5edfe-121f-4cc5-a4dd-75967d0f2810 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:09.665148 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f5edfe-121f-4cc5-a4dd-75967d0f2810 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:09.665424 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f5edfe-121f-4cc5-a4dd-75967d0f2810 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:09.665691 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f5edfe-121f-4cc5-a4dd-75967d0f2810 admin admin] PATH_INFO: `/v3/projects/ed51504401994bf085a4ebbb2383f2ad/users/5be30a560f7841b2ab32f6a4910e2f19/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:09.680296 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f5edfe-121f-4cc5-a4dd-75967d0f2810 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ed51504401994bf085a4ebbb2383f2ad, user_id=5be30a560f7841b2ab32f6a4910e2f19, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:09.682631 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f5edfe-121f-4cc5-a4dd-75967d0f2810 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:09.693411 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:09.699410 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1321/2648] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:04:09 2022] PUT /identity/v3/projects/ed51504401994bf085a4ebbb2383f2ad/users/5be30a560f7841b2ab32f6a4910e2f19/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:09.704607 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32603bdd-bf35-437b-8f94-31026d3c2e59 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:09.704889 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32603bdd-bf35-437b-8f94-31026d3c2e59 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:09.705162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32603bdd-bf35-437b-8f94-31026d3c2e59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:09.724731 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-32603bdd-bf35-437b-8f94-31026d3c2e59 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:09.725960 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-32603bdd-bf35-437b-8f94-31026d3c2e59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:09.745163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:09.784798 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1328/2649] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:09.794401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f5edfe-121f-4cc5-a4dd-75967d0f2810 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:09.796263 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-80f5edfe-121f-4cc5-a4dd-75967d0f2810 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:09.852602 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f5edfe-121f-4cc5-a4dd-75967d0f2810 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:09.856466 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806d5607-c00b-48ff-a455-0e0fb676cb9e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:09.857082 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806d5607-c00b-48ff-a455-0e0fb676cb9e service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:09.857340 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806d5607-c00b-48ff-a455-0e0fb676cb9e service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:09.857614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806d5607-c00b-48ff-a455-0e0fb676cb9e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:09.866371 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806d5607-c00b-48ff-a455-0e0fb676cb9e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:09.867752 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806d5607-c00b-48ff-a455-0e0fb676cb9e service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:09.877409 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1322/2650] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:04:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:04:10.498153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b26d5ea-1d05-4ca1-b1cb-91c3bcfa8dfb None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:10.498435 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b26d5ea-1d05-4ca1-b1cb-91c3bcfa8dfb None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:10.498783 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b26d5ea-1d05-4ca1-b1cb-91c3bcfa8dfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:10.540173 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-8b26d5ea-1d05-4ca1-b1cb-91c3bcfa8dfb None None] MFA Rules not processed for user `5be30a560f7841b2ab32f6a4910e2f19`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:10.541387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-8b26d5ea-1d05-4ca1-b1cb-91c3bcfa8dfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:10.609210 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1329/2651] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:10 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:10.613778 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806d5607-c00b-48ff-a455-0e0fb676cb9e service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:10.623568 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806d5607-c00b-48ff-a455-0e0fb676cb9e service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:10.628661 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9253e9f4-f5fc-4aae-a705-2609f04ca125 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:10.629271 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9253e9f4-f5fc-4aae-a705-2609f04ca125 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:10.629554 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9253e9f4-f5fc-4aae-a705-2609f04ca125 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:10.629830 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9253e9f4-f5fc-4aae-a705-2609f04ca125 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:10.630511 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9253e9f4-f5fc-4aae-a705-2609f04ca125 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:10.632404 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9253e9f4-f5fc-4aae-a705-2609f04ca125 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:10.660844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1323/2652] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:04:10 2022] POST /identity/v3/projects => generated 405 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:04:10.664938 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b26d5ea-1d05-4ca1-b1cb-91c3bcfa8dfb None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:10.676334 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b26d5ea-1d05-4ca1-b1cb-91c3bcfa8dfb None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:10.734021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d80ab76-3dee-424f-b3ed-61cb6c6b3fb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:10.734656 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d80ab76-3dee-424f-b3ed-61cb6c6b3fb4 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:10.734935 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d80ab76-3dee-424f-b3ed-61cb6c6b3fb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:10.735205 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d80ab76-3dee-424f-b3ed-61cb6c6b3fb4 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:10.735823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d80ab76-3dee-424f-b3ed-61cb6c6b3fb4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:10.737735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d80ab76-3dee-424f-b3ed-61cb6c6b3fb4 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:10.758933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1330/2653] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:04:10 2022] POST /identity/v3/users => generated 309 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 01:04:10.763453 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9253e9f4-f5fc-4aae-a705-2609f04ca125 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:10.773546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9253e9f4-f5fc-4aae-a705-2609f04ca125 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:10.778357 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f433cf6-1aee-4bfc-8e1a-9cbf536f94cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:10.778987 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f433cf6-1aee-4bfc-8e1a-9cbf536f94cc admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:10.779262 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f433cf6-1aee-4bfc-8e1a-9cbf536f94cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:10.779533 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f433cf6-1aee-4bfc-8e1a-9cbf536f94cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:10.780169 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f433cf6-1aee-4bfc-8e1a-9cbf536f94cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:10.781733 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f433cf6-1aee-4bfc-8e1a-9cbf536f94cc admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:10.792147 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1324/2654] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:04:10 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:04:10.796982 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d80ab76-3dee-424f-b3ed-61cb6c6b3fb4 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:10.806858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d80ab76-3dee-424f-b3ed-61cb6c6b3fb4 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:10.811989 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7ec8f9-145d-42d0-ac97-74b9987023d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:10.812632 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7ec8f9-145d-42d0-ac97-74b9987023d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:10.812914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7ec8f9-145d-42d0-ac97-74b9987023d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:10.813198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7ec8f9-145d-42d0-ac97-74b9987023d1 admin admin] PATH_INFO: `/v3/projects/97a5c072a2fc4b219c169a45e2e9541c/users/b6041f8a0f014a629ad2eda6785b27e1/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:10.828748 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7ec8f9-145d-42d0-ac97-74b9987023d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=97a5c072a2fc4b219c169a45e2e9541c, user_id=b6041f8a0f014a629ad2eda6785b27e1, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:10.831446 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7ec8f9-145d-42d0-ac97-74b9987023d1 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:10.843928 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1331/2655] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:04:10 2022] PUT /identity/v3/projects/97a5c072a2fc4b219c169a45e2e9541c/users/b6041f8a0f014a629ad2eda6785b27e1/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:10.847942 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f433cf6-1aee-4bfc-8e1a-9cbf536f94cc admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:10.858331 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f433cf6-1aee-4bfc-8e1a-9cbf536f94cc admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:10.913177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5be563-4564-4f21-8223-9a4e8e8d347b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:10.913772 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5be563-4564-4f21-8223-9a4e8e8d347b admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:10.914046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5be563-4564-4f21-8223-9a4e8e8d347b admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:10.914313 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5be563-4564-4f21-8223-9a4e8e8d347b admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:10.915010 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5be563-4564-4f21-8223-9a4e8e8d347b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:10.916756 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5be563-4564-4f21-8223-9a4e8e8d347b admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:10.926924 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1325/2656] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:04:10 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:04:10.931083 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7ec8f9-145d-42d0-ac97-74b9987023d1 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:10.941623 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7ec8f9-145d-42d0-ac97-74b9987023d1 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:10.946145 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e858e71f-f4ab-478a-838a-738ee11ca487 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:10.946815 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e858e71f-f4ab-478a-838a-738ee11ca487 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:10.947093 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e858e71f-f4ab-478a-838a-738ee11ca487 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:10.947361 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e858e71f-f4ab-478a-838a-738ee11ca487 admin admin] PATH_INFO: `/v3/projects/97a5c072a2fc4b219c169a45e2e9541c/users/b6041f8a0f014a629ad2eda6785b27e1/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:10.949313 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e858e71f-f4ab-478a-838a-738ee11ca487 admin admin] RBAC: Authorizing `identity:create_grant(project_id=97a5c072a2fc4b219c169a45e2e9541c, user_id=b6041f8a0f014a629ad2eda6785b27e1, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:10.951957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e858e71f-f4ab-478a-838a-738ee11ca487 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:10.962064 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1332/2657] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:04:10 2022] PUT /identity/v3/projects/97a5c072a2fc4b219c169a45e2e9541c/users/b6041f8a0f014a629ad2eda6785b27e1/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:11.588800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efff731f-403c-47c2-9482-389cad2c1391 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:11.589085 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efff731f-403c-47c2-9482-389cad2c1391 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:11.589556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efff731f-403c-47c2-9482-389cad2c1391 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:11.618500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-efff731f-403c-47c2-9482-389cad2c1391 None None] MFA Rules not processed for user `b6041f8a0f014a629ad2eda6785b27e1`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:11.619973 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-efff731f-403c-47c2-9482-389cad2c1391 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:11.677481 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:11.699529 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1326/2658] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:11 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:12.156408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e858e71f-f4ab-478a-838a-738ee11ca487 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:12.158162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-e858e71f-f4ab-478a-838a-738ee11ca487 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:12.291140 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:12.352055 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e858e71f-f4ab-478a-838a-738ee11ca487 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:12.357647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d33b5d2-ad96-4ccb-b4fd-721219cfc1dc service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8f30116fb5ce43dab69efa8bafd9d049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:12.358260 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d33b5d2-ad96-4ccb-b4fd-721219cfc1dc service placement] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:12.358698 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d33b5d2-ad96-4ccb-b4fd-721219cfc1dc service placement] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:12.358992 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d33b5d2-ad96-4ccb-b4fd-721219cfc1dc service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:12.360751 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-7d33b5d2-ad96-4ccb-b4fd-721219cfc1dc service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:12.373116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d33b5d2-ad96-4ccb-b4fd-721219cfc1dc service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:12.374757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d33b5d2-ad96-4ccb-b4fd-721219cfc1dc service placement] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:12.384986 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1333/2659] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:04:12 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 229 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:04:12.736927 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efff731f-403c-47c2-9482-389cad2c1391 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:12.799519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efff731f-403c-47c2-9482-389cad2c1391 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:12.804054 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47535455-49b0-4438-8e17-2762385d30ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:12.804701 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47535455-49b0-4438-8e17-2762385d30ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:12.804978 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47535455-49b0-4438-8e17-2762385d30ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:12.805246 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47535455-49b0-4438-8e17-2762385d30ab admin admin] PATH_INFO: `/v3/users/5be30a560f7841b2ab32f6a4910e2f19` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:12.806355 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47535455-49b0-4438-8e17-2762385d30ab admin admin] RBAC: Authorizing `identity:delete_user(user_id=5be30a560f7841b2ab32f6a4910e2f19)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:12.808829 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47535455-49b0-4438-8e17-2762385d30ab admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:12.853421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-47535455-49b0-4438-8e17-2762385d30ab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5be30a560f7841b2ab32f6a4910e2f19'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:12.860808 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-47535455-49b0-4438-8e17-2762385d30ab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5be30a560f7841b2ab32f6a4910e2f19'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:12.861420 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-47535455-49b0-4438-8e17-2762385d30ab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5be30a560f7841b2ab32f6a4910e2f19'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:12.862066 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-47535455-49b0-4438-8e17-2762385d30ab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5be30a560f7841b2ab32f6a4910e2f19'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:12.868958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-47535455-49b0-4438-8e17-2762385d30ab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5be30a560f7841b2ab32f6a4910e2f19'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:12.886835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1327/2660] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:04:12 2022] DELETE /identity/v3/users/5be30a560f7841b2ab32f6a4910e2f19 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 01:04:13.052221 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d33b5d2-ad96-4ccb-b4fd-721219cfc1dc service placement] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:13.063614 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d33b5d2-ad96-4ccb-b4fd-721219cfc1dc service placement] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:13.123511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6286c62-7a9d-4f6e-a6a7-be1674f6b487 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:13.124152 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6286c62-7a9d-4f6e-a6a7-be1674f6b487 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:13.124470 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6286c62-7a9d-4f6e-a6a7-be1674f6b487 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:13.124745 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6286c62-7a9d-4f6e-a6a7-be1674f6b487 admin admin] PATH_INFO: `/v3/projects/ed51504401994bf085a4ebbb2383f2ad` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:13.126056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6286c62-7a9d-4f6e-a6a7-be1674f6b487 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ed51504401994bf085a4ebbb2383f2ad)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:13.128031 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6286c62-7a9d-4f6e-a6a7-be1674f6b487 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:13.175858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a6286c62-7a9d-4f6e-a6a7-be1674f6b487 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ed51504401994bf085a4ebbb2383f2ad', 'request_id': 'req-a6286c62-7a9d-4f6e-a6a7-be1674f6b487'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:13.181182 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a6286c62-7a9d-4f6e-a6a7-be1674f6b487 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ed51504401994bf085a4ebbb2383f2ad', 'request_id': 'req-a6286c62-7a9d-4f6e-a6a7-be1674f6b487'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:13.189223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a6286c62-7a9d-4f6e-a6a7-be1674f6b487 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ed51504401994bf085a4ebbb2383f2ad', 'request_id': 'req-a6286c62-7a9d-4f6e-a6a7-be1674f6b487'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:13.200922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a6286c62-7a9d-4f6e-a6a7-be1674f6b487 admin admin] The token cache is being invalidate because project ed51504401994bf085a4ebbb2383f2ad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:04:13.201208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a6286c62-7a9d-4f6e-a6a7-be1674f6b487 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:13.201854 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a6286c62-7a9d-4f6e-a6a7-be1674f6b487 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:13.203331 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1334/2661] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:04:13 2022] DELETE /identity/v3/projects/ed51504401994bf085a4ebbb2383f2ad => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:13.207596 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47535455-49b0-4438-8e17-2762385d30ab admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:13.209798 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-47535455-49b0-4438-8e17-2762385d30ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:13.224874 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47535455-49b0-4438-8e17-2762385d30ab admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:13.281401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4364e3c5-a9ae-4487-a7f3-6e3fc44c7a88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:13.282038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4364e3c5-a9ae-4487-a7f3-6e3fc44c7a88 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:13.282311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4364e3c5-a9ae-4487-a7f3-6e3fc44c7a88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:13.282869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4364e3c5-a9ae-4487-a7f3-6e3fc44c7a88 admin admin] PATH_INFO: `/v3/users/b6041f8a0f014a629ad2eda6785b27e1` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:13.283996 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4364e3c5-a9ae-4487-a7f3-6e3fc44c7a88 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b6041f8a0f014a629ad2eda6785b27e1)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:13.285896 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4364e3c5-a9ae-4487-a7f3-6e3fc44c7a88 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:13.298845 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:13.334482 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-4364e3c5-a9ae-4487-a7f3-6e3fc44c7a88 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b6041f8a0f014a629ad2eda6785b27e1'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:13.341060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-4364e3c5-a9ae-4487-a7f3-6e3fc44c7a88 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b6041f8a0f014a629ad2eda6785b27e1'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:13.341754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-4364e3c5-a9ae-4487-a7f3-6e3fc44c7a88 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b6041f8a0f014a629ad2eda6785b27e1'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:13.342371 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-4364e3c5-a9ae-4487-a7f3-6e3fc44c7a88 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b6041f8a0f014a629ad2eda6785b27e1'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:13.349222 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-4364e3c5-a9ae-4487-a7f3-6e3fc44c7a88 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b6041f8a0f014a629ad2eda6785b27e1'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:13.366838 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1328/2662] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:04:13 2022] DELETE /identity/v3/users/b6041f8a0f014a629ad2eda6785b27e1 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:13.546133 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6286c62-7a9d-4f6e-a6a7-be1674f6b487 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:13.548104 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a6286c62-7a9d-4f6e-a6a7-be1674f6b487 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:13.561310 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6286c62-7a9d-4f6e-a6a7-be1674f6b487 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:13.591614 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:13.624594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb04f77-a4cb-400b-81b3-6a6608fc55f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:13.625254 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb04f77-a4cb-400b-81b3-6a6608fc55f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:13.625532 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb04f77-a4cb-400b-81b3-6a6608fc55f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:13.625813 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb04f77-a4cb-400b-81b3-6a6608fc55f0 admin admin] PATH_INFO: `/v3/projects/97a5c072a2fc4b219c169a45e2e9541c` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:13.626967 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb04f77-a4cb-400b-81b3-6a6608fc55f0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=97a5c072a2fc4b219c169a45e2e9541c)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:13.628898 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb04f77-a4cb-400b-81b3-6a6608fc55f0 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:13.677820 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7eb04f77-a4cb-400b-81b3-6a6608fc55f0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '97a5c072a2fc4b219c169a45e2e9541c', 'request_id': 'req-7eb04f77-a4cb-400b-81b3-6a6608fc55f0'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:13.683472 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7eb04f77-a4cb-400b-81b3-6a6608fc55f0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '97a5c072a2fc4b219c169a45e2e9541c', 'request_id': 'req-7eb04f77-a4cb-400b-81b3-6a6608fc55f0'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:13.691387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7eb04f77-a4cb-400b-81b3-6a6608fc55f0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '97a5c072a2fc4b219c169a45e2e9541c', 'request_id': 'req-7eb04f77-a4cb-400b-81b3-6a6608fc55f0'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:13.703740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7eb04f77-a4cb-400b-81b3-6a6608fc55f0 admin admin] The token cache is being invalidate because project 97a5c072a2fc4b219c169a45e2e9541c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:04:13.704280 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7eb04f77-a4cb-400b-81b3-6a6608fc55f0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:13.704920 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7eb04f77-a4cb-400b-81b3-6a6608fc55f0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:13.706509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1335/2663] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:04:13 2022] DELETE /identity/v3/projects/97a5c072a2fc4b219c169a45e2e9541c => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:13.712201 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7450ce8-dfc1-4539-a45f-e505d9b2a09c None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:13.712469 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7450ce8-dfc1-4539-a45f-e505d9b2a09c None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:13.712756 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7450ce8-dfc1-4539-a45f-e505d9b2a09c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:13.733472 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-b7450ce8-dfc1-4539-a45f-e505d9b2a09c None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:13.734647 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-b7450ce8-dfc1-4539-a45f-e505d9b2a09c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:13.793501 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1329/2664] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:13.801245 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22958ab8-a150-4742-8990-ad322dbea37f None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:13.801556 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22958ab8-a150-4742-8990-ad322dbea37f None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:13.801830 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22958ab8-a150-4742-8990-ad322dbea37f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:13.820997 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-22958ab8-a150-4742-8990-ad322dbea37f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:13.822189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-22958ab8-a150-4742-8990-ad322dbea37f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:13.830780 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1336/2665] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:13.835212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7450ce8-dfc1-4539-a45f-e505d9b2a09c None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:13.845395 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7450ce8-dfc1-4539-a45f-e505d9b2a09c None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:13.849940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65878ea-eac6-40e0-9398-f3eda67a26bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:13.850562 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65878ea-eac6-40e0-9398-f3eda67a26bf admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:13.850839 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65878ea-eac6-40e0-9398-f3eda67a26bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:13.851108 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65878ea-eac6-40e0-9398-f3eda67a26bf admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:13.851752 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65878ea-eac6-40e0-9398-f3eda67a26bf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:04:13.852078 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65878ea-eac6-40e0-9398-f3eda67a26bf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:13.853682 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65878ea-eac6-40e0-9398-f3eda67a26bf admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:13.865033 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1330/2666] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:04:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 01:04:13.869356 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22958ab8-a150-4742-8990-ad322dbea37f None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:13.878564 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22958ab8-a150-4742-8990-ad322dbea37f None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:13.883241 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3606651a-8489-40e6-b318-833137188a93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:13.883835 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3606651a-8489-40e6-b318-833137188a93 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:13.884121 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3606651a-8489-40e6-b318-833137188a93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:13.884398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3606651a-8489-40e6-b318-833137188a93 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:13.885027 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3606651a-8489-40e6-b318-833137188a93 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:13.887496 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3606651a-8489-40e6-b318-833137188a93 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:13.914466 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1337/2667] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:04:13 2022] POST /identity/v3/projects => generated 379 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:04:13.918721 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65878ea-eac6-40e0-9398-f3eda67a26bf admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:13.928797 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65878ea-eac6-40e0-9398-f3eda67a26bf admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:13.983999 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b946d4a-c60c-4b19-9de6-e356ab867782 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:13.984531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b946d4a-c60c-4b19-9de6-e356ab867782 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:13.984813 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b946d4a-c60c-4b19-9de6-e356ab867782 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:13.985085 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b946d4a-c60c-4b19-9de6-e356ab867782 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:13.985758 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b946d4a-c60c-4b19-9de6-e356ab867782 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:13.988003 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b946d4a-c60c-4b19-9de6-e356ab867782 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:14.009756 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1331/2668] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:04:13 2022] POST /identity/v3/users => generated 290 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:04:14.014735 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3606651a-8489-40e6-b318-833137188a93 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:14.025381 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3606651a-8489-40e6-b318-833137188a93 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:14.030639 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92341f63-ce44-4961-b3b7-20bfb6c4cd29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:14.031252 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92341f63-ce44-4961-b3b7-20bfb6c4cd29 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:14.031529 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92341f63-ce44-4961-b3b7-20bfb6c4cd29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:14.031800 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92341f63-ce44-4961-b3b7-20bfb6c4cd29 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:14.032451 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92341f63-ce44-4961-b3b7-20bfb6c4cd29 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:14.034044 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92341f63-ce44-4961-b3b7-20bfb6c4cd29 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:14.044688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1338/2669] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:04:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:04:14.049514 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b946d4a-c60c-4b19-9de6-e356ab867782 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:14.059692 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b946d4a-c60c-4b19-9de6-e356ab867782 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:14.064311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea29a42d-023f-4d47-8148-40a47f13b136 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:14.064899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea29a42d-023f-4d47-8148-40a47f13b136 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:14.065178 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea29a42d-023f-4d47-8148-40a47f13b136 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:14.065447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea29a42d-023f-4d47-8148-40a47f13b136 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:14.066084 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea29a42d-023f-4d47-8148-40a47f13b136 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:14.067704 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea29a42d-023f-4d47-8148-40a47f13b136 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:14.069408 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:14.081432 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1332/2670] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:04:14 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:04:14.086123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92341f63-ce44-4961-b3b7-20bfb6c4cd29 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:14.096622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92341f63-ce44-4961-b3b7-20bfb6c4cd29 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:14.101226 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0fda18-1e40-41ba-8b72-058c8e7066b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:14.101866 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0fda18-1e40-41ba-8b72-058c8e7066b3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:14.102141 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0fda18-1e40-41ba-8b72-058c8e7066b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:14.102410 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0fda18-1e40-41ba-8b72-058c8e7066b3 admin admin] PATH_INFO: `/v3/projects/e5b54cd6570f400eb3cc4b91e4619f64/users/cf14ecce9cbc46ccb183a328c0a529fc/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:14.117264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0fda18-1e40-41ba-8b72-058c8e7066b3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e5b54cd6570f400eb3cc4b91e4619f64, user_id=cf14ecce9cbc46ccb183a328c0a529fc, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:14.119949 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0fda18-1e40-41ba-8b72-058c8e7066b3 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:14.133089 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1339/2671] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:04:14 2022] PUT /identity/v3/projects/e5b54cd6570f400eb3cc4b91e4619f64/users/cf14ecce9cbc46ccb183a328c0a529fc/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:14.143955 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3110ed8-47d4-40da-956e-ba9dddb3d365 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:14.144286 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3110ed8-47d4-40da-956e-ba9dddb3d365 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:14.144581 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3110ed8-47d4-40da-956e-ba9dddb3d365 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:14.171467 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-d3110ed8-47d4-40da-956e-ba9dddb3d365 None None] MFA Rules not processed for user `cf14ecce9cbc46ccb183a328c0a529fc`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:14.172693 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d3110ed8-47d4-40da-956e-ba9dddb3d365 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:14.237107 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1333/2672] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:14 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:14.242115 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0fda18-1e40-41ba-8b72-058c8e7066b3 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:14.298556 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:14.304971 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0fda18-1e40-41ba-8b72-058c8e7066b3 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:14.311219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d6f981-ace4-4458-a11f-69a7201c7759 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:14.311835 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d6f981-ace4-4458-a11f-69a7201c7759 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:14.312137 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d6f981-ace4-4458-a11f-69a7201c7759 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:14.312420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d6f981-ace4-4458-a11f-69a7201c7759 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:14.313084 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d6f981-ace4-4458-a11f-69a7201c7759 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:14.315038 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d6f981-ace4-4458-a11f-69a7201c7759 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:14.343759 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1340/2673] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:04:14 2022] POST /identity/v3/projects => generated 379 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:04:14.347968 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3110ed8-47d4-40da-956e-ba9dddb3d365 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:14.358679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3110ed8-47d4-40da-956e-ba9dddb3d365 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:14.413312 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0566f3d7-dc32-489a-b8f3-aa20a5389913 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:14.413912 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0566f3d7-dc32-489a-b8f3-aa20a5389913 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:14.414206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0566f3d7-dc32-489a-b8f3-aa20a5389913 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:14.414612 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0566f3d7-dc32-489a-b8f3-aa20a5389913 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:14.415355 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0566f3d7-dc32-489a-b8f3-aa20a5389913 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:14.417265 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0566f3d7-dc32-489a-b8f3-aa20a5389913 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:14.438239 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1334/2674] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:04:14 2022] POST /identity/v3/users => generated 296 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:04:14.442375 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d6f981-ace4-4458-a11f-69a7201c7759 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:14.453206 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d6f981-ace4-4458-a11f-69a7201c7759 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:14.457860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251bca69-3192-445e-8eb7-3fe82b20a377 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:14.458745 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251bca69-3192-445e-8eb7-3fe82b20a377 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:14.459024 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251bca69-3192-445e-8eb7-3fe82b20a377 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:14.459293 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251bca69-3192-445e-8eb7-3fe82b20a377 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:14.459930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251bca69-3192-445e-8eb7-3fe82b20a377 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:14.461519 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251bca69-3192-445e-8eb7-3fe82b20a377 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:14.472015 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1341/2675] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:04:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:04:14.476346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0566f3d7-dc32-489a-b8f3-aa20a5389913 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:14.486174 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0566f3d7-dc32-489a-b8f3-aa20a5389913 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:14.490807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d41d425-678c-465b-a298-f316fbba9930 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:14.491387 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d41d425-678c-465b-a298-f316fbba9930 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:14.491664 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d41d425-678c-465b-a298-f316fbba9930 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:14.491936 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d41d425-678c-465b-a298-f316fbba9930 admin admin] PATH_INFO: `/v3/projects/39f1b28de94148f3947124bf616b809f/users/db0f55ac828b462aabb1de4d733bc7e3/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:14.507187 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d41d425-678c-465b-a298-f316fbba9930 admin admin] RBAC: Authorizing `identity:create_grant(project_id=39f1b28de94148f3947124bf616b809f, user_id=db0f55ac828b462aabb1de4d733bc7e3, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:14.509804 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d41d425-678c-465b-a298-f316fbba9930 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:14.524050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1335/2676] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:04:14 2022] PUT /identity/v3/projects/39f1b28de94148f3947124bf616b809f/users/db0f55ac828b462aabb1de4d733bc7e3/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:14.528008 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251bca69-3192-445e-8eb7-3fe82b20a377 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:14.537850 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251bca69-3192-445e-8eb7-3fe82b20a377 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:14.597061 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149d310c-205d-4ce5-aa09-1d6f04afed13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:14.597658 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149d310c-205d-4ce5-aa09-1d6f04afed13 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:14.597933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149d310c-205d-4ce5-aa09-1d6f04afed13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:14.598209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149d310c-205d-4ce5-aa09-1d6f04afed13 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:14.598874 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149d310c-205d-4ce5-aa09-1d6f04afed13 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:14.600446 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149d310c-205d-4ce5-aa09-1d6f04afed13 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:14.610519 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1342/2677] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:04:14 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:04:14.615349 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d41d425-678c-465b-a298-f316fbba9930 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:14.627259 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d41d425-678c-465b-a298-f316fbba9930 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:14.631907 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54a8261-a42a-46e0-82c6-0ef3c990e430 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:14.632549 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54a8261-a42a-46e0-82c6-0ef3c990e430 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:14.632822 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54a8261-a42a-46e0-82c6-0ef3c990e430 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:14.633090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54a8261-a42a-46e0-82c6-0ef3c990e430 admin admin] PATH_INFO: `/v3/projects/39f1b28de94148f3947124bf616b809f/users/db0f55ac828b462aabb1de4d733bc7e3/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:14.634998 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54a8261-a42a-46e0-82c6-0ef3c990e430 admin admin] RBAC: Authorizing `identity:create_grant(project_id=39f1b28de94148f3947124bf616b809f, user_id=db0f55ac828b462aabb1de4d733bc7e3, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:14.637586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54a8261-a42a-46e0-82c6-0ef3c990e430 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:14.647819 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1336/2678] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:04:14 2022] PUT /identity/v3/projects/39f1b28de94148f3947124bf616b809f/users/db0f55ac828b462aabb1de4d733bc7e3/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:14.659427 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f27abf2-a67f-4909-8d6a-d75f653383e6 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:14.659730 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f27abf2-a67f-4909-8d6a-d75f653383e6 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:14.660003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f27abf2-a67f-4909-8d6a-d75f653383e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:14.687905 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-4f27abf2-a67f-4909-8d6a-d75f653383e6 None None] MFA Rules not processed for user `db0f55ac828b462aabb1de4d733bc7e3`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:14.689092 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-4f27abf2-a67f-4909-8d6a-d75f653383e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:14.754125 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1343/2679] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:14 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:14.765251 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54a8261-a42a-46e0-82c6-0ef3c990e430 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:14.767032 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c54a8261-a42a-46e0-82c6-0ef3c990e430 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:14.810611 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:14.826915 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54a8261-a42a-46e0-82c6-0ef3c990e430 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:14.830589 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4951a6-97ce-49b5-8b03-fcc0d5956827 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:14.831183 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4951a6-97ce-49b5-8b03-fcc0d5956827 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:14.831464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4951a6-97ce-49b5-8b03-fcc0d5956827 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:14.831734 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4951a6-97ce-49b5-8b03-fcc0d5956827 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:14.841141 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4951a6-97ce-49b5-8b03-fcc0d5956827 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:14.842616 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4951a6-97ce-49b5-8b03-fcc0d5956827 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:14.852126 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1337/2680] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:04:14 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:04:14.927529 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f27abf2-a67f-4909-8d6a-d75f653383e6 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:14.937057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f27abf2-a67f-4909-8d6a-d75f653383e6 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:14.940632 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea6c96e-0b70-42d2-912a-5357721c11b0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:14.941219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea6c96e-0b70-42d2-912a-5357721c11b0 service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:14.941493 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea6c96e-0b70-42d2-912a-5357721c11b0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:14.941762 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea6c96e-0b70-42d2-912a-5357721c11b0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:14.987437 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:14.998363 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea6c96e-0b70-42d2-912a-5357721c11b0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:14.999902 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea6c96e-0b70-42d2-912a-5357721c11b0 service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:15.008717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1344/2681] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:04:14 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:04:23.892910 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4951a6-97ce-49b5-8b03-fcc0d5956827 service cinder] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:23.953941 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4951a6-97ce-49b5-8b03-fcc0d5956827 service cinder] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:23.958546 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acff46c-03c8-4d43-b554-3a9e52e64170 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:23.959438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acff46c-03c8-4d43-b554-3a9e52e64170 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:23.959714 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acff46c-03c8-4d43-b554-3a9e52e64170 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:23.959984 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acff46c-03c8-4d43-b554-3a9e52e64170 admin admin] PATH_INFO: `/v3/users/cf14ecce9cbc46ccb183a328c0a529fc` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:23.961093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acff46c-03c8-4d43-b554-3a9e52e64170 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cf14ecce9cbc46ccb183a328c0a529fc)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:23.963020 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acff46c-03c8-4d43-b554-3a9e52e64170 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:24.012646 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-3acff46c-03c8-4d43-b554-3a9e52e64170 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cf14ecce9cbc46ccb183a328c0a529fc'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:24.019671 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-3acff46c-03c8-4d43-b554-3a9e52e64170 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cf14ecce9cbc46ccb183a328c0a529fc'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:24.020276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-3acff46c-03c8-4d43-b554-3a9e52e64170 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cf14ecce9cbc46ccb183a328c0a529fc'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:24.020875 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-3acff46c-03c8-4d43-b554-3a9e52e64170 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cf14ecce9cbc46ccb183a328c0a529fc'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:24.028548 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-3acff46c-03c8-4d43-b554-3a9e52e64170 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cf14ecce9cbc46ccb183a328c0a529fc'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:24.045671 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1338/2682] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:04:23 2022] DELETE /identity/v3/users/cf14ecce9cbc46ccb183a328c0a529fc => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:24.050392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcb6d1e-1106-446b-81a8-e88373fa5ce9 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:24.050896 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcb6d1e-1106-446b-81a8-e88373fa5ce9 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:24.050993 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcb6d1e-1106-446b-81a8-e88373fa5ce9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:24.069868 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-0fcb6d1e-1106-446b-81a8-e88373fa5ce9 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:24.071287 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-0fcb6d1e-1106-446b-81a8-e88373fa5ce9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:24.133947 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1345/2683] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:24.143804 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acff46c-03c8-4d43-b554-3a9e52e64170 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:24.145274 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-3acff46c-03c8-4d43-b554-3a9e52e64170 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:24.203687 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acff46c-03c8-4d43-b554-3a9e52e64170 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:24.207353 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f535c82b-511a-4db7-b1fe-5ecc19bbb0b8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:24.207959 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f535c82b-511a-4db7-b1fe-5ecc19bbb0b8 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:24.208239 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f535c82b-511a-4db7-b1fe-5ecc19bbb0b8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:24.208510 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f535c82b-511a-4db7-b1fe-5ecc19bbb0b8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:24.218591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f535c82b-511a-4db7-b1fe-5ecc19bbb0b8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:24.220015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f535c82b-511a-4db7-b1fe-5ecc19bbb0b8 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:24.229998 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1339/2684] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:04:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:04:24.501634 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcb6d1e-1106-446b-81a8-e88373fa5ce9 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:24.503060 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-0fcb6d1e-1106-446b-81a8-e88373fa5ce9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:24.524321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcb6d1e-1106-446b-81a8-e88373fa5ce9 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:24.527209 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f535c82b-511a-4db7-b1fe-5ecc19bbb0b8 service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:24.529361 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3df4274-95be-4ac9-88be-b6940964feb8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:24.530165 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3df4274-95be-4ac9-88be-b6940964feb8 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:24.530328 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3df4274-95be-4ac9-88be-b6940964feb8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:24.530620 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-f535c82b-511a-4db7-b1fe-5ecc19bbb0b8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:24.530767 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3df4274-95be-4ac9-88be-b6940964feb8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:24.532518 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a3df4274-95be-4ac9-88be-b6940964feb8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:24.547789 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:24.552134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3df4274-95be-4ac9-88be-b6940964feb8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:24.553968 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3df4274-95be-4ac9-88be-b6940964feb8 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:24.560999 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f535c82b-511a-4db7-b1fe-5ecc19bbb0b8 service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:24.565869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def1a743-6d1d-45b9-9cd7-9eb136fff3ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:24.566532 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1346/2685] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:04:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:04:24.566660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def1a743-6d1d-45b9-9cd7-9eb136fff3ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:24.566863 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def1a743-6d1d-45b9-9cd7-9eb136fff3ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:24.567135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def1a743-6d1d-45b9-9cd7-9eb136fff3ae admin admin] PATH_INFO: `/v3/projects/e5b54cd6570f400eb3cc4b91e4619f64` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:24.568367 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def1a743-6d1d-45b9-9cd7-9eb136fff3ae admin admin] RBAC: Authorizing `identity:delete_project(project_id=e5b54cd6570f400eb3cc4b91e4619f64)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:24.570401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def1a743-6d1d-45b9-9cd7-9eb136fff3ae admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:24.622416 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-def1a743-6d1d-45b9-9cd7-9eb136fff3ae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e5b54cd6570f400eb3cc4b91e4619f64', 'request_id': 'req-def1a743-6d1d-45b9-9cd7-9eb136fff3ae'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:24.628630 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-def1a743-6d1d-45b9-9cd7-9eb136fff3ae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e5b54cd6570f400eb3cc4b91e4619f64', 'request_id': 'req-def1a743-6d1d-45b9-9cd7-9eb136fff3ae'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:24.639186 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-def1a743-6d1d-45b9-9cd7-9eb136fff3ae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e5b54cd6570f400eb3cc4b91e4619f64', 'request_id': 'req-def1a743-6d1d-45b9-9cd7-9eb136fff3ae'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:24.651813 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-def1a743-6d1d-45b9-9cd7-9eb136fff3ae admin admin] The token cache is being invalidate because project e5b54cd6570f400eb3cc4b91e4619f64 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:04:24.652214 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-def1a743-6d1d-45b9-9cd7-9eb136fff3ae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:24.652994 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-def1a743-6d1d-45b9-9cd7-9eb136fff3ae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:24.654609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1340/2686] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:04:24 2022] DELETE /identity/v3/projects/e5b54cd6570f400eb3cc4b91e4619f64 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:24.658520 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3df4274-95be-4ac9-88be-b6940964feb8 service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:24.668477 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3df4274-95be-4ac9-88be-b6940964feb8 service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:24.725130 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a674f09-ddba-42cd-bf8e-d9e5feaa242a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:24.725766 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a674f09-ddba-42cd-bf8e-d9e5feaa242a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:24.726061 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a674f09-ddba-42cd-bf8e-d9e5feaa242a admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:24.726333 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a674f09-ddba-42cd-bf8e-d9e5feaa242a admin admin] PATH_INFO: `/v3/users/db0f55ac828b462aabb1de4d733bc7e3` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:24.727490 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a674f09-ddba-42cd-bf8e-d9e5feaa242a admin admin] RBAC: Authorizing `identity:delete_user(user_id=db0f55ac828b462aabb1de4d733bc7e3)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:24.729471 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a674f09-ddba-42cd-bf8e-d9e5feaa242a admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:24.779390 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5a674f09-ddba-42cd-bf8e-d9e5feaa242a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'db0f55ac828b462aabb1de4d733bc7e3'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:24.787780 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5a674f09-ddba-42cd-bf8e-d9e5feaa242a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'db0f55ac828b462aabb1de4d733bc7e3'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:24.800969 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5a674f09-ddba-42cd-bf8e-d9e5feaa242a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'db0f55ac828b462aabb1de4d733bc7e3'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:24.801685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5a674f09-ddba-42cd-bf8e-d9e5feaa242a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'db0f55ac828b462aabb1de4d733bc7e3'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:24.807138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:24.809718 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5a674f09-ddba-42cd-bf8e-d9e5feaa242a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'db0f55ac828b462aabb1de4d733bc7e3'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:24.817288 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1347/2687] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:04:24 2022] DELETE /identity/v3/users/db0f55ac828b462aabb1de4d733bc7e3 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:25.291515 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def1a743-6d1d-45b9-9cd7-9eb136fff3ae admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:25.293783 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-def1a743-6d1d-45b9-9cd7-9eb136fff3ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:25.313307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def1a743-6d1d-45b9-9cd7-9eb136fff3ae admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:25.375898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05ac762-6791-4921-b972-3e9f61dc4693 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:25.376532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05ac762-6791-4921-b972-3e9f61dc4693 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:25.376924 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05ac762-6791-4921-b972-3e9f61dc4693 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:25.377197 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05ac762-6791-4921-b972-3e9f61dc4693 admin admin] PATH_INFO: `/v3/projects/39f1b28de94148f3947124bf616b809f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:25.378312 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05ac762-6791-4921-b972-3e9f61dc4693 admin admin] RBAC: Authorizing `identity:delete_project(project_id=39f1b28de94148f3947124bf616b809f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:25.380217 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05ac762-6791-4921-b972-3e9f61dc4693 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:25.428927 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-f05ac762-6791-4921-b972-3e9f61dc4693 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '39f1b28de94148f3947124bf616b809f', 'request_id': 'req-f05ac762-6791-4921-b972-3e9f61dc4693'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:25.435012 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-f05ac762-6791-4921-b972-3e9f61dc4693 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '39f1b28de94148f3947124bf616b809f', 'request_id': 'req-f05ac762-6791-4921-b972-3e9f61dc4693'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:25.443454 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-f05ac762-6791-4921-b972-3e9f61dc4693 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '39f1b28de94148f3947124bf616b809f', 'request_id': 'req-f05ac762-6791-4921-b972-3e9f61dc4693'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:25.455006 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-f05ac762-6791-4921-b972-3e9f61dc4693 admin admin] The token cache is being invalidate because project 39f1b28de94148f3947124bf616b809f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:04:25.455360 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-f05ac762-6791-4921-b972-3e9f61dc4693 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:25.456448 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-f05ac762-6791-4921-b972-3e9f61dc4693 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:25.457675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1341/2688] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:04:25 2022] DELETE /identity/v3/projects/39f1b28de94148f3947124bf616b809f => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:25.463815 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de1e848-b6ac-40b9-aa89-ad0af72a43d3 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:25.464311 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de1e848-b6ac-40b9-aa89-ad0af72a43d3 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:25.464311 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de1e848-b6ac-40b9-aa89-ad0af72a43d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:25.484854 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-2de1e848-b6ac-40b9-aa89-ad0af72a43d3 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:25.486162 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-2de1e848-b6ac-40b9-aa89-ad0af72a43d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:25.567473 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1348/2689] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:25.575618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536a5946-fc56-40e7-a213-aaca5c913903 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:25.575898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536a5946-fc56-40e7-a213-aaca5c913903 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:25.576170 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536a5946-fc56-40e7-a213-aaca5c913903 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:25.602025 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-536a5946-fc56-40e7-a213-aaca5c913903 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:25.603568 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-536a5946-fc56-40e7-a213-aaca5c913903 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:25.612786 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1342/2690] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:25.617366 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de1e848-b6ac-40b9-aa89-ad0af72a43d3 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:25.628668 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de1e848-b6ac-40b9-aa89-ad0af72a43d3 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:25.633609 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d83412-aa6e-43e2-834c-9cc2d63f2eef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:25.634501 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d83412-aa6e-43e2-834c-9cc2d63f2eef admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:25.634659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d83412-aa6e-43e2-834c-9cc2d63f2eef admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:25.634844 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d83412-aa6e-43e2-834c-9cc2d63f2eef admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:25.635521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d83412-aa6e-43e2-834c-9cc2d63f2eef admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:04:25.636205 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d83412-aa6e-43e2-834c-9cc2d63f2eef admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:25.637535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d83412-aa6e-43e2-834c-9cc2d63f2eef admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:25.654694 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1349/2691] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:04:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 01:04:25.656633 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536a5946-fc56-40e7-a213-aaca5c913903 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:25.668157 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536a5946-fc56-40e7-a213-aaca5c913903 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:25.674231 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe49ef7-5c5d-429c-94a8-380dc9d5502f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:25.674882 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe49ef7-5c5d-429c-94a8-380dc9d5502f admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:25.675162 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe49ef7-5c5d-429c-94a8-380dc9d5502f admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:25.675432 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe49ef7-5c5d-429c-94a8-380dc9d5502f admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:25.676119 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe49ef7-5c5d-429c-94a8-380dc9d5502f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:25.678006 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe49ef7-5c5d-429c-94a8-380dc9d5502f admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:25.684954 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:25.714818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1343/2692] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:04:25 2022] POST /identity/v3/projects => generated 383 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:04:25.718662 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d83412-aa6e-43e2-834c-9cc2d63f2eef admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:25.729956 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d83412-aa6e-43e2-834c-9cc2d63f2eef admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:25.794065 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d36d3a8-5d9a-4a52-8477-e570b9c5aef0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:25.796105 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d36d3a8-5d9a-4a52-8477-e570b9c5aef0 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:25.796421 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d36d3a8-5d9a-4a52-8477-e570b9c5aef0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:25.796699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d36d3a8-5d9a-4a52-8477-e570b9c5aef0 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:25.797398 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d36d3a8-5d9a-4a52-8477-e570b9c5aef0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:25.799537 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d36d3a8-5d9a-4a52-8477-e570b9c5aef0 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:25.824031 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1350/2693] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:04:25 2022] POST /identity/v3/users => generated 292 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:04:25.828734 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe49ef7-5c5d-429c-94a8-380dc9d5502f admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:25.839902 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe49ef7-5c5d-429c-94a8-380dc9d5502f admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:25.845196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292c9992-704c-4af0-a46f-d28d63d54f27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:25.845801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292c9992-704c-4af0-a46f-d28d63d54f27 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:25.846071 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292c9992-704c-4af0-a46f-d28d63d54f27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:25.846336 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292c9992-704c-4af0-a46f-d28d63d54f27 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:25.847004 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292c9992-704c-4af0-a46f-d28d63d54f27 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:25.848575 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292c9992-704c-4af0-a46f-d28d63d54f27 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:25.859019 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1344/2694] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:04:25 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:04:25.863322 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d36d3a8-5d9a-4a52-8477-e570b9c5aef0 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:25.874628 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d36d3a8-5d9a-4a52-8477-e570b9c5aef0 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:25.879425 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a646448e-1252-43c3-8069-74c14991551a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:25.880029 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a646448e-1252-43c3-8069-74c14991551a admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:25.880310 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a646448e-1252-43c3-8069-74c14991551a admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:25.880587 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a646448e-1252-43c3-8069-74c14991551a admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:25.881232 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a646448e-1252-43c3-8069-74c14991551a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:25.882881 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a646448e-1252-43c3-8069-74c14991551a admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:25.895121 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1351/2695] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:04:25 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:04:25.899542 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292c9992-704c-4af0-a46f-d28d63d54f27 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:25.910051 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292c9992-704c-4af0-a46f-d28d63d54f27 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:25.914671 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9283f0-0d21-4b7c-a222-a00983cc380d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:25.915321 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9283f0-0d21-4b7c-a222-a00983cc380d admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:25.915598 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9283f0-0d21-4b7c-a222-a00983cc380d admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:25.915870 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9283f0-0d21-4b7c-a222-a00983cc380d admin admin] PATH_INFO: `/v3/projects/840f016e848a4abcae77b4f61ddc4f31/users/0a33da5e823844829f792046acd7f9bb/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:25.930374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9283f0-0d21-4b7c-a222-a00983cc380d admin admin] RBAC: Authorizing `identity:create_grant(project_id=840f016e848a4abcae77b4f61ddc4f31, user_id=0a33da5e823844829f792046acd7f9bb, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:25.933031 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9283f0-0d21-4b7c-a222-a00983cc380d admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:25.946736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1345/2696] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:04:25 2022] PUT /identity/v3/projects/840f016e848a4abcae77b4f61ddc4f31/users/0a33da5e823844829f792046acd7f9bb/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 01:04:25.957756 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3b926a-6ee7-482e-8b02-518c5cc307ac None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:25.958167 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3b926a-6ee7-482e-8b02-518c5cc307ac None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:25.958578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3b926a-6ee7-482e-8b02-518c5cc307ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:25.986880 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-0b3b926a-6ee7-482e-8b02-518c5cc307ac None None] MFA Rules not processed for user `0a33da5e823844829f792046acd7f9bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:25.988038 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-0b3b926a-6ee7-482e-8b02-518c5cc307ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:26.036082 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:26.054505 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1352/2697] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:25 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 01:04:26.059159 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9283f0-0d21-4b7c-a222-a00983cc380d admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:26.143337 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9283f0-0d21-4b7c-a222-a00983cc380d admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:26.149431 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a29de0-d631-4bbb-aed5-13973b32ea03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:26.150038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a29de0-d631-4bbb-aed5-13973b32ea03 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:26.150312 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a29de0-d631-4bbb-aed5-13973b32ea03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:26.150614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a29de0-d631-4bbb-aed5-13973b32ea03 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:26.151254 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a29de0-d631-4bbb-aed5-13973b32ea03 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:26.153204 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a29de0-d631-4bbb-aed5-13973b32ea03 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:26.188310 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1346/2698] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:04:26 2022] POST /identity/v3/projects => generated 381 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 01:04:26.189788 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3b926a-6ee7-482e-8b02-518c5cc307ac None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:26.200870 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3b926a-6ee7-482e-8b02-518c5cc307ac None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:26.264740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b9fd03-6a2e-4deb-8762-9cb1610af561 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:26.265272 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b9fd03-6a2e-4deb-8762-9cb1610af561 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:26.265556 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b9fd03-6a2e-4deb-8762-9cb1610af561 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:26.265821 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b9fd03-6a2e-4deb-8762-9cb1610af561 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:26.266486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b9fd03-6a2e-4deb-8762-9cb1610af561 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:26.268446 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b9fd03-6a2e-4deb-8762-9cb1610af561 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:26.309916 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1353/2699] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:04:26 2022] POST /identity/v3/users => generated 297 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 01:04:26.311448 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a29de0-d631-4bbb-aed5-13973b32ea03 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:26.324284 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a29de0-d631-4bbb-aed5-13973b32ea03 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:26.332604 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3663ce2e-3f05-4ffd-ae03-75b40bb6aff3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:26.334167 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3663ce2e-3f05-4ffd-ae03-75b40bb6aff3 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:26.334795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3663ce2e-3f05-4ffd-ae03-75b40bb6aff3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:26.336328 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3663ce2e-3f05-4ffd-ae03-75b40bb6aff3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:26.337171 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3663ce2e-3f05-4ffd-ae03-75b40bb6aff3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:26.339487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3663ce2e-3f05-4ffd-ae03-75b40bb6aff3 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:26.361955 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1347/2700] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:04:26 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:04:26.363234 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b9fd03-6a2e-4deb-8762-9cb1610af561 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:26.374444 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b9fd03-6a2e-4deb-8762-9cb1610af561 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:26.384972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49da1896-662f-4d33-a820-481e9524c1fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:26.385987 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49da1896-662f-4d33-a820-481e9524c1fc admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:26.386391 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49da1896-662f-4d33-a820-481e9524c1fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:26.386811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49da1896-662f-4d33-a820-481e9524c1fc admin admin] PATH_INFO: `/v3/projects/40c79c1162294999a4842aa17d393aca/users/0e917f59bca94e7687b2d20d0788caa2/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:26.403848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49da1896-662f-4d33-a820-481e9524c1fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=40c79c1162294999a4842aa17d393aca, user_id=0e917f59bca94e7687b2d20d0788caa2, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:26.406825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49da1896-662f-4d33-a820-481e9524c1fc admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:26.421775 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1354/2701] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:04:26 2022] PUT /identity/v3/projects/40c79c1162294999a4842aa17d393aca/users/0e917f59bca94e7687b2d20d0788caa2/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:26.425650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3663ce2e-3f05-4ffd-ae03-75b40bb6aff3 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:26.436743 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3663ce2e-3f05-4ffd-ae03-75b40bb6aff3 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:26.503602 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699cb63e-1f74-4acb-9c7d-55838f8d18b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:26.504227 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699cb63e-1f74-4acb-9c7d-55838f8d18b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:26.504506 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699cb63e-1f74-4acb-9c7d-55838f8d18b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:26.504779 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699cb63e-1f74-4acb-9c7d-55838f8d18b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:26.505439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699cb63e-1f74-4acb-9c7d-55838f8d18b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:26.507437 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699cb63e-1f74-4acb-9c7d-55838f8d18b3 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:26.525587 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1348/2702] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:04:26 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:04:26.528644 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49da1896-662f-4d33-a820-481e9524c1fc admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:26.540544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49da1896-662f-4d33-a820-481e9524c1fc admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:26.546504 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07377ab2-1b72-48cf-b6b4-3a5c5a7bdb78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:26.547189 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07377ab2-1b72-48cf-b6b4-3a5c5a7bdb78 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:26.547467 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07377ab2-1b72-48cf-b6b4-3a5c5a7bdb78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:26.547740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07377ab2-1b72-48cf-b6b4-3a5c5a7bdb78 admin admin] PATH_INFO: `/v3/projects/40c79c1162294999a4842aa17d393aca/users/0e917f59bca94e7687b2d20d0788caa2/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:26.549959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07377ab2-1b72-48cf-b6b4-3a5c5a7bdb78 admin admin] RBAC: Authorizing `identity:create_grant(project_id=40c79c1162294999a4842aa17d393aca, user_id=0e917f59bca94e7687b2d20d0788caa2, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:26.552731 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07377ab2-1b72-48cf-b6b4-3a5c5a7bdb78 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:26.565390 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1355/2703] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:04:26 2022] PUT /identity/v3/projects/40c79c1162294999a4842aa17d393aca/users/0e917f59bca94e7687b2d20d0788caa2/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:26.577444 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37ec378-e675-4a86-b4ff-de7d7e3d2f17 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:26.577705 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37ec378-e675-4a86-b4ff-de7d7e3d2f17 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:26.577985 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37ec378-e675-4a86-b4ff-de7d7e3d2f17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:26.609999 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-d37ec378-e675-4a86-b4ff-de7d7e3d2f17 None None] MFA Rules not processed for user `0e917f59bca94e7687b2d20d0788caa2`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:26.611275 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d37ec378-e675-4a86-b4ff-de7d7e3d2f17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:26.623092 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:26.702214 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1349/2704] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:26 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:26.713707 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07377ab2-1b72-48cf-b6b4-3a5c5a7bdb78 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:26.715711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-07377ab2-1b72-48cf-b6b4-3a5c5a7bdb78 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:26.782880 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07377ab2-1b72-48cf-b6b4-3a5c5a7bdb78 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:26.786767 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de0534b-a0bd-4f3a-99e7-4d9df338d8fd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:26.787387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de0534b-a0bd-4f3a-99e7-4d9df338d8fd service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:26.787775 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de0534b-a0bd-4f3a-99e7-4d9df338d8fd service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:26.787944 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de0534b-a0bd-4f3a-99e7-4d9df338d8fd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:26.797667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de0534b-a0bd-4f3a-99e7-4d9df338d8fd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:26.799149 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de0534b-a0bd-4f3a-99e7-4d9df338d8fd service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:26.808770 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1356/2705] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:04:26 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:04:26.922307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37ec378-e675-4a86-b4ff-de7d7e3d2f17 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:26.932057 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37ec378-e675-4a86-b4ff-de7d7e3d2f17 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:26.936262 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b772cb13-ad9a-4a9e-8af9-a31b851f3861 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:26.936841 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b772cb13-ad9a-4a9e-8af9-a31b851f3861 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:26.937141 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b772cb13-ad9a-4a9e-8af9-a31b851f3861 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:26.937415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b772cb13-ad9a-4a9e-8af9-a31b851f3861 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:26.998194 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b772cb13-ad9a-4a9e-8af9-a31b851f3861 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:26.999677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b772cb13-ad9a-4a9e-8af9-a31b851f3861 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:27.011112 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1350/2706] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:04:26 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:04:27.962204 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de0534b-a0bd-4f3a-99e7-4d9df338d8fd service cinder] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:27.963675 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-7de0534b-a0bd-4f3a-99e7-4d9df338d8fd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:28.013686 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:28.031017 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de0534b-a0bd-4f3a-99e7-4d9df338d8fd service cinder] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:28.035634 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006c11b3-1df9-4dcc-a3b1-4d1d9d18f5fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:28.036273 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006c11b3-1df9-4dcc-a3b1-4d1d9d18f5fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:28.036556 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006c11b3-1df9-4dcc-a3b1-4d1d9d18f5fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:28.036833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006c11b3-1df9-4dcc-a3b1-4d1d9d18f5fe admin admin] PATH_INFO: `/v3/users/294ae98a8492468893beb3b8c2870d5e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:28.037975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006c11b3-1df9-4dcc-a3b1-4d1d9d18f5fe admin admin] RBAC: Authorizing `identity:delete_user(user_id=294ae98a8492468893beb3b8c2870d5e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:28.040062 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006c11b3-1df9-4dcc-a3b1-4d1d9d18f5fe admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:28.091639 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-006c11b3-1df9-4dcc-a3b1-4d1d9d18f5fe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '294ae98a8492468893beb3b8c2870d5e'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:28.099353 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-006c11b3-1df9-4dcc-a3b1-4d1d9d18f5fe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '294ae98a8492468893beb3b8c2870d5e'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:28.111959 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-006c11b3-1df9-4dcc-a3b1-4d1d9d18f5fe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '294ae98a8492468893beb3b8c2870d5e'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:28.112612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-006c11b3-1df9-4dcc-a3b1-4d1d9d18f5fe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '294ae98a8492468893beb3b8c2870d5e'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:28.119825 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-006c11b3-1df9-4dcc-a3b1-4d1d9d18f5fe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '294ae98a8492468893beb3b8c2870d5e'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:28.125670 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1357/2707] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:04:27 2022] DELETE /identity/v3/users/294ae98a8492468893beb3b8c2870d5e => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:28.295624 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b772cb13-ad9a-4a9e-8af9-a31b851f3861 service cinder] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:28.308997 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b772cb13-ad9a-4a9e-8af9-a31b851f3861 service cinder] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:28.377502 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03520abf-a344-4819-8ff6-db6ea11655c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:28.378365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03520abf-a344-4819-8ff6-db6ea11655c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:28.378674 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03520abf-a344-4819-8ff6-db6ea11655c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:28.378955 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03520abf-a344-4819-8ff6-db6ea11655c3 admin admin] PATH_INFO: `/v3/projects/505bd70415a04f53a64645eb7959a5fb` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:28.380097 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03520abf-a344-4819-8ff6-db6ea11655c3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=505bd70415a04f53a64645eb7959a5fb)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:28.381983 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03520abf-a344-4819-8ff6-db6ea11655c3 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:28.451802 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:28.453586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-03520abf-a344-4819-8ff6-db6ea11655c3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '505bd70415a04f53a64645eb7959a5fb', 'request_id': 'req-03520abf-a344-4819-8ff6-db6ea11655c3'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:28.465164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-03520abf-a344-4819-8ff6-db6ea11655c3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '505bd70415a04f53a64645eb7959a5fb', 'request_id': 'req-03520abf-a344-4819-8ff6-db6ea11655c3'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:28.473210 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-03520abf-a344-4819-8ff6-db6ea11655c3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '505bd70415a04f53a64645eb7959a5fb', 'request_id': 'req-03520abf-a344-4819-8ff6-db6ea11655c3'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:28.485871 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-03520abf-a344-4819-8ff6-db6ea11655c3 admin admin] The token cache is being invalidate because project 505bd70415a04f53a64645eb7959a5fb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:04:28.486169 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-03520abf-a344-4819-8ff6-db6ea11655c3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:28.486847 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-03520abf-a344-4819-8ff6-db6ea11655c3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:28.488329 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1351/2708] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:04:28 2022] DELETE /identity/v3/projects/505bd70415a04f53a64645eb7959a5fb => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:28.493448 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7398588e-04b1-4ca8-add4-e79966053046 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:28.493875 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7398588e-04b1-4ca8-add4-e79966053046 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:28.494226 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7398588e-04b1-4ca8-add4-e79966053046 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:28.521482 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-7398588e-04b1-4ca8-add4-e79966053046 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:28.522880 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-7398588e-04b1-4ca8-add4-e79966053046 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:28.589124 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1358/2709] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:28.597487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb8d178-e701-41da-aafa-ef4d655b577c None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:28.597648 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb8d178-e701-41da-aafa-ef4d655b577c None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:28.597835 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb8d178-e701-41da-aafa-ef4d655b577c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:28.618141 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-5cb8d178-e701-41da-aafa-ef4d655b577c None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:28.619806 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-5cb8d178-e701-41da-aafa-ef4d655b577c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:28.630834 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1352/2710] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:28.639807 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7398588e-04b1-4ca8-add4-e79966053046 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:28.649300 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7398588e-04b1-4ca8-add4-e79966053046 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:28.653852 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fe884b-0179-4427-bd86-05f54bce42fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:28.654447 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fe884b-0179-4427-bd86-05f54bce42fc admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:28.654750 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fe884b-0179-4427-bd86-05f54bce42fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:28.655020 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fe884b-0179-4427-bd86-05f54bce42fc admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:28.655747 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18fe884b-0179-4427-bd86-05f54bce42fc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:04:28.656045 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18fe884b-0179-4427-bd86-05f54bce42fc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:28.657727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18fe884b-0179-4427-bd86-05f54bce42fc admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:28.669853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1359/2711] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:04:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 18 01:04:28.674313 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb8d178-e701-41da-aafa-ef4d655b577c None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:28.683727 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb8d178-e701-41da-aafa-ef4d655b577c None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:28.688987 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e6813e-4939-4ebb-8266-dd77690cc886 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:28.689556 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e6813e-4939-4ebb-8266-dd77690cc886 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:28.689836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e6813e-4939-4ebb-8266-dd77690cc886 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:28.690105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e6813e-4939-4ebb-8266-dd77690cc886 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:28.690756 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e6813e-4939-4ebb-8266-dd77690cc886 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:28.692618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e6813e-4939-4ebb-8266-dd77690cc886 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:28.720467 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1353/2712] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:04:28 2022] POST /identity/v3/projects => generated 375 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:04:28.724831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fe884b-0179-4427-bd86-05f54bce42fc admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:28.735494 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fe884b-0179-4427-bd86-05f54bce42fc admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:28.781608 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:28.792509 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cdabad2-494b-47a7-84bc-a4f1c0e6f358 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:28.794076 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdabad2-494b-47a7-84bc-a4f1c0e6f358 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:28.794348 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdabad2-494b-47a7-84bc-a4f1c0e6f358 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:28.794647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdabad2-494b-47a7-84bc-a4f1c0e6f358 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:28.795280 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cdabad2-494b-47a7-84bc-a4f1c0e6f358 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:28.797283 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cdabad2-494b-47a7-84bc-a4f1c0e6f358 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:28.820051 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1360/2713] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:04:28 2022] POST /identity/v3/users => generated 288 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 01:04:28.824488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e6813e-4939-4ebb-8266-dd77690cc886 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:28.835621 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e6813e-4939-4ebb-8266-dd77690cc886 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:28.840346 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872b05ae-b737-4a68-ac9c-8f9675b0640e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:28.840953 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872b05ae-b737-4a68-ac9c-8f9675b0640e admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:28.841271 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872b05ae-b737-4a68-ac9c-8f9675b0640e admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:28.841542 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872b05ae-b737-4a68-ac9c-8f9675b0640e admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:28.842206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872b05ae-b737-4a68-ac9c-8f9675b0640e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:28.843840 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872b05ae-b737-4a68-ac9c-8f9675b0640e admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:28.854250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1354/2714] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:04:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:04:28.858831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cdabad2-494b-47a7-84bc-a4f1c0e6f358 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:28.868984 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cdabad2-494b-47a7-84bc-a4f1c0e6f358 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:28.874307 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b97b7f7-1d69-4fe7-add4-85269ed83893 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:28.874940 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b97b7f7-1d69-4fe7-add4-85269ed83893 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:28.875222 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b97b7f7-1d69-4fe7-add4-85269ed83893 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:28.875546 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b97b7f7-1d69-4fe7-add4-85269ed83893 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:28.876179 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b97b7f7-1d69-4fe7-add4-85269ed83893 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:28.878260 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b97b7f7-1d69-4fe7-add4-85269ed83893 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:28.890104 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1361/2715] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:04:28 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:04:28.894318 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872b05ae-b737-4a68-ac9c-8f9675b0640e admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:28.904259 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872b05ae-b737-4a68-ac9c-8f9675b0640e admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:28.909232 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69cea55-80bb-420f-8b87-76824069290a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:28.910095 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69cea55-80bb-420f-8b87-76824069290a admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:28.910365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69cea55-80bb-420f-8b87-76824069290a admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:28.910671 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69cea55-80bb-420f-8b87-76824069290a admin admin] PATH_INFO: `/v3/projects/39509a7e0ca64f94913b5fbf10c6ec9c/users/bd2fa2deb0a24aff9d8b7ce97b6e461b/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:28.925993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69cea55-80bb-420f-8b87-76824069290a admin admin] RBAC: Authorizing `identity:create_grant(project_id=39509a7e0ca64f94913b5fbf10c6ec9c, user_id=bd2fa2deb0a24aff9d8b7ce97b6e461b, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:28.929007 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69cea55-80bb-420f-8b87-76824069290a admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:28.943736 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1355/2716] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:04:28 2022] PUT /identity/v3/projects/39509a7e0ca64f94913b5fbf10c6ec9c/users/bd2fa2deb0a24aff9d8b7ce97b6e461b/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 01:04:28.948816 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7ad17c-a0e7-4e21-8d5b-66531b7e363c None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:28.949290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7ad17c-a0e7-4e21-8d5b-66531b7e363c None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:28.949678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7ad17c-a0e7-4e21-8d5b-66531b7e363c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:28.969754 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-2e7ad17c-a0e7-4e21-8d5b-66531b7e363c None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:28.970984 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-2e7ad17c-a0e7-4e21-8d5b-66531b7e363c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:29.029941 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1362/2717] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:29.039678 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69cea55-80bb-420f-8b87-76824069290a admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:29.041347 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d69cea55-80bb-420f-8b87-76824069290a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:29.100153 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69cea55-80bb-420f-8b87-76824069290a admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:29.103817 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b455d4b5-5204-44d3-979b-1fc8901b2c05 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:29.104474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b455d4b5-5204-44d3-979b-1fc8901b2c05 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:29.104697 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b455d4b5-5204-44d3-979b-1fc8901b2c05 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:29.104969 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b455d4b5-5204-44d3-979b-1fc8901b2c05 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:29.114205 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b455d4b5-5204-44d3-979b-1fc8901b2c05 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:29.115660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b455d4b5-5204-44d3-979b-1fc8901b2c05 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:29.125977 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1356/2718] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:04:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:04:29.665798 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7ad17c-a0e7-4e21-8d5b-66531b7e363c None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:29.667270 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-2e7ad17c-a0e7-4e21-8d5b-66531b7e363c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:29.679013 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7ad17c-a0e7-4e21-8d5b-66531b7e363c None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:29.683806 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f85c25-af45-4f0f-918f-0024a12eb988 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:29.684507 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f85c25-af45-4f0f-918f-0024a12eb988 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:29.684791 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f85c25-af45-4f0f-918f-0024a12eb988 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:29.685064 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f85c25-af45-4f0f-918f-0024a12eb988 admin admin] PATH_INFO: `/v3/users/0a33da5e823844829f792046acd7f9bb` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:29.686226 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f85c25-af45-4f0f-918f-0024a12eb988 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0a33da5e823844829f792046acd7f9bb)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:29.688197 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f85c25-af45-4f0f-918f-0024a12eb988 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:29.743986 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a4f85c25-af45-4f0f-918f-0024a12eb988 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0a33da5e823844829f792046acd7f9bb'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:29.755541 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a4f85c25-af45-4f0f-918f-0024a12eb988 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0a33da5e823844829f792046acd7f9bb'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:29.769344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a4f85c25-af45-4f0f-918f-0024a12eb988 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0a33da5e823844829f792046acd7f9bb'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:29.770071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a4f85c25-af45-4f0f-918f-0024a12eb988 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0a33da5e823844829f792046acd7f9bb'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:29.780975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a4f85c25-af45-4f0f-918f-0024a12eb988 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0a33da5e823844829f792046acd7f9bb'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:29.787926 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1363/2719] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:04:29 2022] DELETE /identity/v3/users/0a33da5e823844829f792046acd7f9bb => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:29.792636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dd797e-80d1-4d1a-83ad-f67a29c3128e None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:29.793279 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dd797e-80d1-4d1a-83ad-f67a29c3128e None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:29.794426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dd797e-80d1-4d1a-83ad-f67a29c3128e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:29.814515 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-f9dd797e-80d1-4d1a-83ad-f67a29c3128e None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:29.815879 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-f9dd797e-80d1-4d1a-83ad-f67a29c3128e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:29.898275 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1357/2720] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:29.905120 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f85c25-af45-4f0f-918f-0024a12eb988 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:29.906610 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a4f85c25-af45-4f0f-918f-0024a12eb988 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:29.967823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:29.981036 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f85c25-af45-4f0f-918f-0024a12eb988 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:29.987411 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6fc290-b678-4ed8-b83d-37208fa1f853 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:29.988517 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6fc290-b678-4ed8-b83d-37208fa1f853 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:29.988994 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6fc290-b678-4ed8-b83d-37208fa1f853 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:29.989496 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6fc290-b678-4ed8-b83d-37208fa1f853 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:30.000613 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6fc290-b678-4ed8-b83d-37208fa1f853 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:30.002346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6fc290-b678-4ed8-b83d-37208fa1f853 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:30.013572 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1364/2721] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:04:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:04:30.480820 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dd797e-80d1-4d1a-83ad-f67a29c3128e None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:30.482240 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-f9dd797e-80d1-4d1a-83ad-f67a29c3128e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:30.495745 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dd797e-80d1-4d1a-83ad-f67a29c3128e None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:30.500713 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cde10a-d2ae-49bb-964c-da230b3b39b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:30.501349 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cde10a-d2ae-49bb-964c-da230b3b39b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:30.501652 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cde10a-d2ae-49bb-964c-da230b3b39b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:30.501876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cde10a-d2ae-49bb-964c-da230b3b39b5 admin admin] PATH_INFO: `/v3/projects/840f016e848a4abcae77b4f61ddc4f31` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:30.503846 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5cde10a-d2ae-49bb-964c-da230b3b39b5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=840f016e848a4abcae77b4f61ddc4f31)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:30.505733 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5cde10a-d2ae-49bb-964c-da230b3b39b5 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:30.518251 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=94,keystone:DELETE=3,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:30.574095 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-d5cde10a-d2ae-49bb-964c-da230b3b39b5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '840f016e848a4abcae77b4f61ddc4f31', 'request_id': 'req-d5cde10a-d2ae-49bb-964c-da230b3b39b5'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:30.581250 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-d5cde10a-d2ae-49bb-964c-da230b3b39b5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '840f016e848a4abcae77b4f61ddc4f31', 'request_id': 'req-d5cde10a-d2ae-49bb-964c-da230b3b39b5'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:30.593359 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-d5cde10a-d2ae-49bb-964c-da230b3b39b5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '840f016e848a4abcae77b4f61ddc4f31', 'request_id': 'req-d5cde10a-d2ae-49bb-964c-da230b3b39b5'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:30.607210 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-d5cde10a-d2ae-49bb-964c-da230b3b39b5 admin admin] The token cache is being invalidate because project 840f016e848a4abcae77b4f61ddc4f31 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:04:30.607889 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-d5cde10a-d2ae-49bb-964c-da230b3b39b5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:30.608177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-d5cde10a-d2ae-49bb-964c-da230b3b39b5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:30.610499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1358/2722] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:04:30 2022] DELETE /identity/v3/projects/840f016e848a4abcae77b4f61ddc4f31 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:30.614403 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6fc290-b678-4ed8-b83d-37208fa1f853 service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:30.625615 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6fc290-b678-4ed8-b83d-37208fa1f853 service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:30.732575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e472b2c-790d-4239-aa03-42556ad92e8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:30.733251 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e472b2c-790d-4239-aa03-42556ad92e8e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:30.746446 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e472b2c-790d-4239-aa03-42556ad92e8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:30.748124 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e472b2c-790d-4239-aa03-42556ad92e8e admin admin] PATH_INFO: `/v3/users/0e917f59bca94e7687b2d20d0788caa2` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:30.749687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e472b2c-790d-4239-aa03-42556ad92e8e admin admin] RBAC: Authorizing `identity:delete_user(user_id=0e917f59bca94e7687b2d20d0788caa2)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:30.755485 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e472b2c-790d-4239-aa03-42556ad92e8e admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:30.826755 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-4e472b2c-790d-4239-aa03-42556ad92e8e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0e917f59bca94e7687b2d20d0788caa2'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:30.837476 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-4e472b2c-790d-4239-aa03-42556ad92e8e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0e917f59bca94e7687b2d20d0788caa2'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:30.853209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-4e472b2c-790d-4239-aa03-42556ad92e8e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0e917f59bca94e7687b2d20d0788caa2'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:30.856633 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-4e472b2c-790d-4239-aa03-42556ad92e8e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0e917f59bca94e7687b2d20d0788caa2'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:30.864313 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-4e472b2c-790d-4239-aa03-42556ad92e8e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0e917f59bca94e7687b2d20d0788caa2'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:30.870153 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1365/2723] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:04:30 2022] DELETE /identity/v3/users/0e917f59bca94e7687b2d20d0788caa2 => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:31.364269 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cde10a-d2ae-49bb-964c-da230b3b39b5 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:31.365898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d5cde10a-d2ae-49bb-964c-da230b3b39b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:31.382830 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cde10a-d2ae-49bb-964c-da230b3b39b5 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:31.465258 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abdf7d4-a72a-4d55-9fae-e7da5e0be197 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:31.465794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abdf7d4-a72a-4d55-9fae-e7da5e0be197 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:31.466021 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abdf7d4-a72a-4d55-9fae-e7da5e0be197 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:31.466396 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abdf7d4-a72a-4d55-9fae-e7da5e0be197 admin admin] PATH_INFO: `/v3/projects/40c79c1162294999a4842aa17d393aca` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:31.467620 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abdf7d4-a72a-4d55-9fae-e7da5e0be197 admin admin] RBAC: Authorizing `identity:delete_project(project_id=40c79c1162294999a4842aa17d393aca)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:31.469776 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abdf7d4-a72a-4d55-9fae-e7da5e0be197 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:31.537289 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6abdf7d4-a72a-4d55-9fae-e7da5e0be197 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '40c79c1162294999a4842aa17d393aca', 'request_id': 'req-6abdf7d4-a72a-4d55-9fae-e7da5e0be197'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:31.544171 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6abdf7d4-a72a-4d55-9fae-e7da5e0be197 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '40c79c1162294999a4842aa17d393aca', 'request_id': 'req-6abdf7d4-a72a-4d55-9fae-e7da5e0be197'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:31.553959 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6abdf7d4-a72a-4d55-9fae-e7da5e0be197 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '40c79c1162294999a4842aa17d393aca', 'request_id': 'req-6abdf7d4-a72a-4d55-9fae-e7da5e0be197'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:31.580355 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6abdf7d4-a72a-4d55-9fae-e7da5e0be197 admin admin] The token cache is being invalidate because project 40c79c1162294999a4842aa17d393aca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:04:31.580867 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6abdf7d4-a72a-4d55-9fae-e7da5e0be197 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:31.582495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-6abdf7d4-a72a-4d55-9fae-e7da5e0be197 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:31.583812 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1359/2724] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:04:31 2022] DELETE /identity/v3/projects/40c79c1162294999a4842aa17d393aca => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:31.588835 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce4c2f2-2618-4c57-a78d-fef2059a0138 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:31.589748 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce4c2f2-2618-4c57-a78d-fef2059a0138 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:31.590785 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce4c2f2-2618-4c57-a78d-fef2059a0138 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:31.611795 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-2ce4c2f2-2618-4c57-a78d-fef2059a0138 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:31.613429 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-2ce4c2f2-2618-4c57-a78d-fef2059a0138 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:31.697780 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1366/2725] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:31.706140 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50f6f51-bd4b-402e-9721-887c19d03cf1 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:31.706419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50f6f51-bd4b-402e-9721-887c19d03cf1 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:31.706746 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50f6f51-bd4b-402e-9721-887c19d03cf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:31.732439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-c50f6f51-bd4b-402e-9721-887c19d03cf1 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:31.733676 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c50f6f51-bd4b-402e-9721-887c19d03cf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:31.746519 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1360/2726] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:31.752227 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce4c2f2-2618-4c57-a78d-fef2059a0138 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:31.764393 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce4c2f2-2618-4c57-a78d-fef2059a0138 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:31.769706 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de496859-2169-444f-b41e-3d66b34a013a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:31.770661 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de496859-2169-444f-b41e-3d66b34a013a admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:31.771160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de496859-2169-444f-b41e-3d66b34a013a admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:31.771658 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de496859-2169-444f-b41e-3d66b34a013a admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:31.772575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de496859-2169-444f-b41e-3d66b34a013a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:04:31.773109 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de496859-2169-444f-b41e-3d66b34a013a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:31.775378 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de496859-2169-444f-b41e-3d66b34a013a admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:31.777400 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:31.796392 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1367/2727] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:04:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 01:04:31.798376 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50f6f51-bd4b-402e-9721-887c19d03cf1 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:31.811379 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50f6f51-bd4b-402e-9721-887c19d03cf1 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:31.819218 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c008ee-fdd5-4efb-be76-2d6ff12091d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:31.819845 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c008ee-fdd5-4efb-be76-2d6ff12091d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:31.820119 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c008ee-fdd5-4efb-be76-2d6ff12091d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:31.820387 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c008ee-fdd5-4efb-be76-2d6ff12091d4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:31.821026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c008ee-fdd5-4efb-be76-2d6ff12091d4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:31.823231 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c008ee-fdd5-4efb-be76-2d6ff12091d4 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:31.849213 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:31.862920 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1361/2728] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:04:31 2022] POST /identity/v3/projects => generated 373 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:04:31.864029 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de496859-2169-444f-b41e-3d66b34a013a admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:31.880534 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de496859-2169-444f-b41e-3d66b34a013a admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:31.958010 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4299aa-133d-4dff-b64a-0694cb791031 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:31.958742 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4299aa-133d-4dff-b64a-0694cb791031 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:31.959845 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4299aa-133d-4dff-b64a-0694cb791031 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:31.959845 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4299aa-133d-4dff-b64a-0694cb791031 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:31.960558 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4299aa-133d-4dff-b64a-0694cb791031 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:31.962652 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4299aa-133d-4dff-b64a-0694cb791031 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:31.991709 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1368/2729] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:04:31 2022] POST /identity/v3/users => generated 287 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:04:31.995759 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c008ee-fdd5-4efb-be76-2d6ff12091d4 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:32.007887 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c008ee-fdd5-4efb-be76-2d6ff12091d4 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:32.012830 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5013806c-8347-46b0-b25b-a4a27b725075 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:32.013437 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5013806c-8347-46b0-b25b-a4a27b725075 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:32.013713 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5013806c-8347-46b0-b25b-a4a27b725075 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:32.013986 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5013806c-8347-46b0-b25b-a4a27b725075 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:32.014669 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5013806c-8347-46b0-b25b-a4a27b725075 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:32.016441 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5013806c-8347-46b0-b25b-a4a27b725075 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:32.032670 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4299aa-133d-4dff-b64a-0694cb791031 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:32.033096 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1362/2730] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:04:31 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:04:32.043179 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4299aa-133d-4dff-b64a-0694cb791031 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:32.049873 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339c571b-cde8-40a3-a78d-9f3a62000e4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:32.050751 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339c571b-cde8-40a3-a78d-9f3a62000e4f admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:32.050751 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339c571b-cde8-40a3-a78d-9f3a62000e4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:32.051077 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339c571b-cde8-40a3-a78d-9f3a62000e4f admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:32.051667 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339c571b-cde8-40a3-a78d-9f3a62000e4f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:32.053316 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339c571b-cde8-40a3-a78d-9f3a62000e4f admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:32.068199 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1369/2731] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:04:32 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:04:32.073129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5013806c-8347-46b0-b25b-a4a27b725075 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:32.085125 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5013806c-8347-46b0-b25b-a4a27b725075 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:32.091627 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08a883a-4d23-4d40-b61b-cc6ba6ed422c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:32.092305 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08a883a-4d23-4d40-b61b-cc6ba6ed422c admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:32.092573 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08a883a-4d23-4d40-b61b-cc6ba6ed422c admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:32.092850 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08a883a-4d23-4d40-b61b-cc6ba6ed422c admin admin] PATH_INFO: `/v3/projects/2ac4ac0c047c40bb87e591c4e981c965/users/45c6ce9d651c4487b1be6c20a7c04cc4/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:32.109283 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08a883a-4d23-4d40-b61b-cc6ba6ed422c admin admin] RBAC: Authorizing `identity:create_grant(project_id=2ac4ac0c047c40bb87e591c4e981c965, user_id=45c6ce9d651c4487b1be6c20a7c04cc4, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:32.112235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08a883a-4d23-4d40-b61b-cc6ba6ed422c admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:32.139908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1363/2732] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:04:32 2022] PUT /identity/v3/projects/2ac4ac0c047c40bb87e591c4e981c965/users/45c6ce9d651c4487b1be6c20a7c04cc4/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:32.140216 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd01bc6c-0e13-4f19-973e-2ee93dda89eb None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:32.140216 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd01bc6c-0e13-4f19-973e-2ee93dda89eb None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:32.140216 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd01bc6c-0e13-4f19-973e-2ee93dda89eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:32.161330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-bd01bc6c-0e13-4f19-973e-2ee93dda89eb None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:32.162645 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-bd01bc6c-0e13-4f19-973e-2ee93dda89eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:32.232491 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1370/2733] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:32.246656 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08a883a-4d23-4d40-b61b-cc6ba6ed422c admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:32.248395 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c08a883a-4d23-4d40-b61b-cc6ba6ed422c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:32.329004 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08a883a-4d23-4d40-b61b-cc6ba6ed422c admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:32.334979 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56480aa-e4ae-4c1b-b4cf-688552159c07 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:32.336152 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56480aa-e4ae-4c1b-b4cf-688552159c07 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:32.338561 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56480aa-e4ae-4c1b-b4cf-688552159c07 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:32.339951 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56480aa-e4ae-4c1b-b4cf-688552159c07 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:32.353404 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c56480aa-e4ae-4c1b-b4cf-688552159c07 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:32.355123 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c56480aa-e4ae-4c1b-b4cf-688552159c07 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:32.372080 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1364/2734] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:04:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:04:33.171865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc66dfb-6c40-44a6-b7f9-ee745114c874 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:33.171865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc66dfb-6c40-44a6-b7f9-ee745114c874 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:33.171865 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc66dfb-6c40-44a6-b7f9-ee745114c874 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:33.203150 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-1fc66dfb-6c40-44a6-b7f9-ee745114c874 None None] MFA Rules not processed for user `45c6ce9d651c4487b1be6c20a7c04cc4`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:33.204605 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-1fc66dfb-6c40-44a6-b7f9-ee745114c874 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:33.297418 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1371/2735] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:33 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:33.298605 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56480aa-e4ae-4c1b-b4cf-688552159c07 service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:33.308560 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56480aa-e4ae-4c1b-b4cf-688552159c07 service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:33.313374 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a7c878-1950-4f0f-a347-3d27974779cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:33.313990 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a7c878-1950-4f0f-a347-3d27974779cf admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:33.314266 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a7c878-1950-4f0f-a347-3d27974779cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:33.314581 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a7c878-1950-4f0f-a347-3d27974779cf admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:33.315219 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a7c878-1950-4f0f-a347-3d27974779cf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:33.317107 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a7c878-1950-4f0f-a347-3d27974779cf admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:33.353187 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1365/2736] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:04:33 2022] POST /identity/v3/projects => generated 373 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:04:33.357517 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc66dfb-6c40-44a6-b7f9-ee745114c874 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:33.363449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:33.373647 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc66dfb-6c40-44a6-b7f9-ee745114c874 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:33.469709 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66695b2-5c80-4a6c-8e05-2830af78928c None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:33.470920 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66695b2-5c80-4a6c-8e05-2830af78928c None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:33.471214 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66695b2-5c80-4a6c-8e05-2830af78928c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:33.472108 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695c40a5-afe4-4c61-a4fb-39233c0a1380 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:33.472724 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695c40a5-afe4-4c61-a4fb-39233c0a1380 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:33.473002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695c40a5-afe4-4c61-a4fb-39233c0a1380 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:33.473271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695c40a5-afe4-4c61-a4fb-39233c0a1380 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:33.473897 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695c40a5-afe4-4c61-a4fb-39233c0a1380 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:33.475836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695c40a5-afe4-4c61-a4fb-39233c0a1380 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:33.498256 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1372/2737] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:04:33 2022] POST /identity/v3/users => generated 293 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 01:04:33.502549 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695c40a5-afe4-4c61-a4fb-39233c0a1380 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:33.516050 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695c40a5-afe4-4c61-a4fb-39233c0a1380 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:33.522073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-d66695b2-5c80-4a6c-8e05-2830af78928c None None] MFA Rules not processed for user `bd2fa2deb0a24aff9d8b7ce97b6e461b`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:33.523344 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96da068-819e-412e-99ee-7a191df38d99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:33.524039 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96da068-819e-412e-99ee-7a191df38d99 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:33.524386 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96da068-819e-412e-99ee-7a191df38d99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:33.524724 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96da068-819e-412e-99ee-7a191df38d99 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:33.525681 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d66695b2-5c80-4a6c-8e05-2830af78928c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:33.525858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96da068-819e-412e-99ee-7a191df38d99 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:33.529403 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96da068-819e-412e-99ee-7a191df38d99 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:33.543908 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1373/2738] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:04:33 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 18 01:04:33.546270 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96da068-819e-412e-99ee-7a191df38d99 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:33.563703 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96da068-819e-412e-99ee-7a191df38d99 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:33.568833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec3b127-ff2e-47c5-8bb7-219edd5bb4ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:33.570508 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec3b127-ff2e-47c5-8bb7-219edd5bb4ba admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:33.570843 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec3b127-ff2e-47c5-8bb7-219edd5bb4ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:33.571122 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec3b127-ff2e-47c5-8bb7-219edd5bb4ba admin admin] PATH_INFO: `/v3/projects/173237a736854bf189e8a9b3503ebd66/users/c9dd2f29d5b1433fb4db7438141721f9/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:33.593247 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec3b127-ff2e-47c5-8bb7-219edd5bb4ba admin admin] RBAC: Authorizing `identity:create_grant(project_id=173237a736854bf189e8a9b3503ebd66, user_id=c9dd2f29d5b1433fb4db7438141721f9, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:33.596109 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec3b127-ff2e-47c5-8bb7-219edd5bb4ba admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:33.611752 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1366/2739] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:33 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 01:04:33.627020 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66695b2-5c80-4a6c-8e05-2830af78928c None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:33.628714 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d66695b2-5c80-4a6c-8e05-2830af78928c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:33.643349 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1374/2740] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:04:33 2022] PUT /identity/v3/projects/173237a736854bf189e8a9b3503ebd66/users/c9dd2f29d5b1433fb4db7438141721f9/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 01:04:33.647884 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec3b127-ff2e-47c5-8bb7-219edd5bb4ba admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:33.678354 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:33.720060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66695b2-5c80-4a6c-8e05-2830af78928c None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:33.720338 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec3b127-ff2e-47c5-8bb7-219edd5bb4ba admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:33.729888 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d9d27f-d5ec-47c3-bdd0-5c737a891e74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:33.730561 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d9d27f-d5ec-47c3-bdd0-5c737a891e74 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:33.731420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d9d27f-d5ec-47c3-bdd0-5c737a891e74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:33.731666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d9d27f-d5ec-47c3-bdd0-5c737a891e74 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:33.732330 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d9d27f-d5ec-47c3-bdd0-5c737a891e74 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:33.733940 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d9d27f-d5ec-47c3-bdd0-5c737a891e74 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:33.744837 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1375/2741] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:04:33 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:04:33.749178 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d9d27f-d5ec-47c3-bdd0-5c737a891e74 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:33.759067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d9d27f-d5ec-47c3-bdd0-5c737a891e74 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:33.764103 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86fd064-3252-49b5-a3ca-5439f037e2d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:33.764832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86fd064-3252-49b5-a3ca-5439f037e2d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:33.765214 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86fd064-3252-49b5-a3ca-5439f037e2d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:33.765533 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86fd064-3252-49b5-a3ca-5439f037e2d5 admin admin] PATH_INFO: `/v3/projects/173237a736854bf189e8a9b3503ebd66/users/c9dd2f29d5b1433fb4db7438141721f9/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:33.767586 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86fd064-3252-49b5-a3ca-5439f037e2d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=173237a736854bf189e8a9b3503ebd66, user_id=c9dd2f29d5b1433fb4db7438141721f9, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:33.770336 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86fd064-3252-49b5-a3ca-5439f037e2d5 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:33.783218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1376/2742] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:04:33 2022] PUT /identity/v3/projects/173237a736854bf189e8a9b3503ebd66/users/c9dd2f29d5b1433fb4db7438141721f9/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 01:04:33.804728 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cedc2add-ad3e-4c95-9144-49ff9c1fa6a4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:33.805461 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedc2add-ad3e-4c95-9144-49ff9c1fa6a4 service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:33.805806 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedc2add-ad3e-4c95-9144-49ff9c1fa6a4 service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:33.806150 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedc2add-ad3e-4c95-9144-49ff9c1fa6a4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:33.875567 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cedc2add-ad3e-4c95-9144-49ff9c1fa6a4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:33.877331 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cedc2add-ad3e-4c95-9144-49ff9c1fa6a4 service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:33.887268 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1367/2743] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:04:33 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 261 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 01:04:34.556195 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd479a7-c860-48e7-a344-517234865e01 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:34.556966 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd479a7-c860-48e7-a344-517234865e01 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:34.556966 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd479a7-c860-48e7-a344-517234865e01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:34.604405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-5bd479a7-c860-48e7-a344-517234865e01 None None] MFA Rules not processed for user `c9dd2f29d5b1433fb4db7438141721f9`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:34.606214 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-5bd479a7-c860-48e7-a344-517234865e01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:34.670939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:34.740099 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1377/2744] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:34 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:34.752404 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cedc2add-ad3e-4c95-9144-49ff9c1fa6a4 service nova] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:34.753980 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-cedc2add-ad3e-4c95-9144-49ff9c1fa6a4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:34.837141 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cedc2add-ad3e-4c95-9144-49ff9c1fa6a4 service nova] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:34.874136 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:34.908618 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4376a76f-431b-456b-a2da-ed16da7e6ab8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:34.909768 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4376a76f-431b-456b-a2da-ed16da7e6ab8 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:34.910223 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4376a76f-431b-456b-a2da-ed16da7e6ab8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:34.910694 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4376a76f-431b-456b-a2da-ed16da7e6ab8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:34.995187 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4376a76f-431b-456b-a2da-ed16da7e6ab8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:34.997334 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4376a76f-431b-456b-a2da-ed16da7e6ab8 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:35.007235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1368/2745] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:04:34 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 255 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:04:35.493464 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684b4fd7-6708-41c8-8eef-91bf6582f322 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:35.493749 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684b4fd7-6708-41c8-8eef-91bf6582f322 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:35.494031 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684b4fd7-6708-41c8-8eef-91bf6582f322 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:35.514396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-684b4fd7-6708-41c8-8eef-91bf6582f322 None None] MFA Rules not processed for user `bd2fa2deb0a24aff9d8b7ce97b6e461b`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:35.515652 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-684b4fd7-6708-41c8-8eef-91bf6582f322 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:35.573957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1378/2746] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:35 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:36.442697 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4376a76f-431b-456b-a2da-ed16da7e6ab8 service cinder] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:36.444781 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-4376a76f-431b-456b-a2da-ed16da7e6ab8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:36.460211 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4376a76f-431b-456b-a2da-ed16da7e6ab8 service cinder] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:36.466093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e951da5d-25be-493e-8cf5-2e1efce3ae4a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:36.466846 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e951da5d-25be-493e-8cf5-2e1efce3ae4a service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:36.467196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e951da5d-25be-493e-8cf5-2e1efce3ae4a service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:36.467606 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e951da5d-25be-493e-8cf5-2e1efce3ae4a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:36.482063 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e951da5d-25be-493e-8cf5-2e1efce3ae4a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:36.484239 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e951da5d-25be-493e-8cf5-2e1efce3ae4a service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:36.498806 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1369/2747] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:04:36 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:04:45.567137 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=41 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:46.490941 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=42 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:50.068622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684b4fd7-6708-41c8-8eef-91bf6582f322 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:50.186385 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684b4fd7-6708-41c8-8eef-91bf6582f322 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:50.190973 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a32f15-fd18-4c87-8cf9-2735e52893e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:50.191608 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a32f15-fd18-4c87-8cf9-2735e52893e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:50.191889 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a32f15-fd18-4c87-8cf9-2735e52893e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:50.192165 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a32f15-fd18-4c87-8cf9-2735e52893e9 admin admin] PATH_INFO: `/v3/users/45c6ce9d651c4487b1be6c20a7c04cc4` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:50.193328 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a32f15-fd18-4c87-8cf9-2735e52893e9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=45c6ce9d651c4487b1be6c20a7c04cc4)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:50.195281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a32f15-fd18-4c87-8cf9-2735e52893e9 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:50.249217 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-43a32f15-fd18-4c87-8cf9-2735e52893e9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '45c6ce9d651c4487b1be6c20a7c04cc4'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:50.258057 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-43a32f15-fd18-4c87-8cf9-2735e52893e9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '45c6ce9d651c4487b1be6c20a7c04cc4'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:50.272030 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-43a32f15-fd18-4c87-8cf9-2735e52893e9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '45c6ce9d651c4487b1be6c20a7c04cc4'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:50.272770 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-43a32f15-fd18-4c87-8cf9-2735e52893e9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '45c6ce9d651c4487b1be6c20a7c04cc4'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:50.287901 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-43a32f15-fd18-4c87-8cf9-2735e52893e9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '45c6ce9d651c4487b1be6c20a7c04cc4'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:50.294759 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1379/2748] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:04:50 2022] DELETE /identity/v3/users/45c6ce9d651c4487b1be6c20a7c04cc4 => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:50.474982 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e951da5d-25be-493e-8cf5-2e1efce3ae4a service cinder] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:50.510332 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e951da5d-25be-493e-8cf5-2e1efce3ae4a service cinder] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:50.569538 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0877a7db-2250-4f72-9a7d-0641909b9fa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:50.570186 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0877a7db-2250-4f72-9a7d-0641909b9fa5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:50.570510 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0877a7db-2250-4f72-9a7d-0641909b9fa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:50.570783 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0877a7db-2250-4f72-9a7d-0641909b9fa5 admin admin] PATH_INFO: `/v3/projects/2ac4ac0c047c40bb87e591c4e981c965` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:50.571901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0877a7db-2250-4f72-9a7d-0641909b9fa5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2ac4ac0c047c40bb87e591c4e981c965)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:50.573815 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0877a7db-2250-4f72-9a7d-0641909b9fa5 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:50.619457 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0877a7db-2250-4f72-9a7d-0641909b9fa5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2ac4ac0c047c40bb87e591c4e981c965', 'request_id': 'req-0877a7db-2250-4f72-9a7d-0641909b9fa5'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:50.625995 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0877a7db-2250-4f72-9a7d-0641909b9fa5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2ac4ac0c047c40bb87e591c4e981c965', 'request_id': 'req-0877a7db-2250-4f72-9a7d-0641909b9fa5'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:50.633277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0877a7db-2250-4f72-9a7d-0641909b9fa5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2ac4ac0c047c40bb87e591c4e981c965', 'request_id': 'req-0877a7db-2250-4f72-9a7d-0641909b9fa5'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:50.656145 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0877a7db-2250-4f72-9a7d-0641909b9fa5 admin admin] The token cache is being invalidate because project 2ac4ac0c047c40bb87e591c4e981c965 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:04:50.656426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0877a7db-2250-4f72-9a7d-0641909b9fa5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:50.657046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0877a7db-2250-4f72-9a7d-0641909b9fa5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:50.658514 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1370/2749] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:04:50 2022] DELETE /identity/v3/projects/2ac4ac0c047c40bb87e591c4e981c965 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:50.662786 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a32f15-fd18-4c87-8cf9-2735e52893e9 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:50.664187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-43a32f15-fd18-4c87-8cf9-2735e52893e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:50.687510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a32f15-fd18-4c87-8cf9-2735e52893e9 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:50.712592 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:50.744606 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1a1aaf-ff1e-429e-ac58-d0276499f96e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:50.745539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1a1aaf-ff1e-429e-ac58-d0276499f96e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:50.746019 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1a1aaf-ff1e-429e-ac58-d0276499f96e admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:50.747053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1a1aaf-ff1e-429e-ac58-d0276499f96e admin admin] PATH_INFO: `/v3/users/c9dd2f29d5b1433fb4db7438141721f9` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:50.749029 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1a1aaf-ff1e-429e-ac58-d0276499f96e admin admin] RBAC: Authorizing `identity:delete_user(user_id=c9dd2f29d5b1433fb4db7438141721f9)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:50.752133 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1a1aaf-ff1e-429e-ac58-d0276499f96e admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:50.800071 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2e1a1aaf-ff1e-429e-ac58-d0276499f96e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c9dd2f29d5b1433fb4db7438141721f9'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:50.808527 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2e1a1aaf-ff1e-429e-ac58-d0276499f96e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c9dd2f29d5b1433fb4db7438141721f9'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:50.821065 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2e1a1aaf-ff1e-429e-ac58-d0276499f96e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c9dd2f29d5b1433fb4db7438141721f9'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:50.821740 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2e1a1aaf-ff1e-429e-ac58-d0276499f96e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c9dd2f29d5b1433fb4db7438141721f9'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:50.828741 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-2e1a1aaf-ff1e-429e-ac58-d0276499f96e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c9dd2f29d5b1433fb4db7438141721f9'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:50.834798 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1380/2750] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:04:50 2022] DELETE /identity/v3/users/c9dd2f29d5b1433fb4db7438141721f9 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:50.998854 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0877a7db-2250-4f72-9a7d-0641909b9fa5 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:51.000400 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-0877a7db-2250-4f72-9a7d-0641909b9fa5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:51.015207 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0877a7db-2250-4f72-9a7d-0641909b9fa5 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:51.065032 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:51.072987 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad1b2f1-af15-41e2-b06b-55b5c4978ed3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:51.073895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad1b2f1-af15-41e2-b06b-55b5c4978ed3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:51.074391 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad1b2f1-af15-41e2-b06b-55b5c4978ed3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:51.075293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad1b2f1-af15-41e2-b06b-55b5c4978ed3 admin admin] PATH_INFO: `/v3/projects/173237a736854bf189e8a9b3503ebd66` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:51.077203 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad1b2f1-af15-41e2-b06b-55b5c4978ed3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=173237a736854bf189e8a9b3503ebd66)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:51.079164 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad1b2f1-af15-41e2-b06b-55b5c4978ed3 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:51.130863 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0ad1b2f1-af15-41e2-b06b-55b5c4978ed3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '173237a736854bf189e8a9b3503ebd66', 'request_id': 'req-0ad1b2f1-af15-41e2-b06b-55b5c4978ed3'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:51.136394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0ad1b2f1-af15-41e2-b06b-55b5c4978ed3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '173237a736854bf189e8a9b3503ebd66', 'request_id': 'req-0ad1b2f1-af15-41e2-b06b-55b5c4978ed3'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:51.144675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0ad1b2f1-af15-41e2-b06b-55b5c4978ed3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '173237a736854bf189e8a9b3503ebd66', 'request_id': 'req-0ad1b2f1-af15-41e2-b06b-55b5c4978ed3'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:51.155774 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0ad1b2f1-af15-41e2-b06b-55b5c4978ed3 admin admin] The token cache is being invalidate because project 173237a736854bf189e8a9b3503ebd66 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:04:51.156053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0ad1b2f1-af15-41e2-b06b-55b5c4978ed3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:51.156707 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-0ad1b2f1-af15-41e2-b06b-55b5c4978ed3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:04:51.158189 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1371/2751] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:04:50 2022] DELETE /identity/v3/projects/173237a736854bf189e8a9b3503ebd66 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:51.163461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3478a539-078d-4a58-b753-7fe1a6d698b6 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:51.163981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3478a539-078d-4a58-b753-7fe1a6d698b6 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:51.164085 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3478a539-078d-4a58-b753-7fe1a6d698b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:51.221465 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-3478a539-078d-4a58-b753-7fe1a6d698b6 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:51.222704 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3478a539-078d-4a58-b753-7fe1a6d698b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:51.280132 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1381/2752] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:51.288047 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98dfe6ff-d1f4-4a0d-81e5-5d6dbb0eb58b None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:51.288331 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98dfe6ff-d1f4-4a0d-81e5-5d6dbb0eb58b None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:51.288605 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98dfe6ff-d1f4-4a0d-81e5-5d6dbb0eb58b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:51.309049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-98dfe6ff-d1f4-4a0d-81e5-5d6dbb0eb58b None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:51.310844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-98dfe6ff-d1f4-4a0d-81e5-5d6dbb0eb58b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:51.318981 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1372/2753] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:51.323421 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3478a539-078d-4a58-b753-7fe1a6d698b6 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:51.333136 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3478a539-078d-4a58-b753-7fe1a6d698b6 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:51.338086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0541b22c-b346-4531-b405-23fec6475f8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:51.338718 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0541b22c-b346-4531-b405-23fec6475f8c admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:51.338998 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0541b22c-b346-4531-b405-23fec6475f8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:51.339271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0541b22c-b346-4531-b405-23fec6475f8c admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:51.339922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0541b22c-b346-4531-b405-23fec6475f8c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:04:51.340226 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0541b22c-b346-4531-b405-23fec6475f8c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:51.341829 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0541b22c-b346-4531-b405-23fec6475f8c admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:51.353642 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1382/2754] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:04:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 01:04:51.358662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98dfe6ff-d1f4-4a0d-81e5-5d6dbb0eb58b None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:51.368148 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98dfe6ff-d1f4-4a0d-81e5-5d6dbb0eb58b None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:51.373523 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b87346a-4638-46c8-937d-b7038b7db589 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:51.374187 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b87346a-4638-46c8-937d-b7038b7db589 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:51.374488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b87346a-4638-46c8-937d-b7038b7db589 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:51.374792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b87346a-4638-46c8-937d-b7038b7db589 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:51.375424 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b87346a-4638-46c8-937d-b7038b7db589 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:51.377316 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b87346a-4638-46c8-937d-b7038b7db589 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:51.411329 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1373/2755] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:04:51 2022] POST /identity/v3/projects => generated 377 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:04:51.412474 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0541b22c-b346-4531-b405-23fec6475f8c admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:51.419448 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:04:51.423385 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0541b22c-b346-4531-b405-23fec6475f8c admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:51.485103 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ea67d0-56fb-4e84-bc5e-d56b073a7509 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:51.485687 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ea67d0-56fb-4e84-bc5e-d56b073a7509 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:51.485969 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ea67d0-56fb-4e84-bc5e-d56b073a7509 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:51.486242 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ea67d0-56fb-4e84-bc5e-d56b073a7509 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:51.486915 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ea67d0-56fb-4e84-bc5e-d56b073a7509 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:51.488823 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ea67d0-56fb-4e84-bc5e-d56b073a7509 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:51.510110 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1383/2756] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:04:51 2022] POST /identity/v3/users => generated 289 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 01:04:51.514113 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b87346a-4638-46c8-937d-b7038b7db589 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:51.525010 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b87346a-4638-46c8-937d-b7038b7db589 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:51.529815 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb65f1f-9f91-4955-99f6-9170c0fce475 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:51.530500 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb65f1f-9f91-4955-99f6-9170c0fce475 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:51.530722 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb65f1f-9f91-4955-99f6-9170c0fce475 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:51.530992 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb65f1f-9f91-4955-99f6-9170c0fce475 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:51.531638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb65f1f-9f91-4955-99f6-9170c0fce475 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:51.533230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb65f1f-9f91-4955-99f6-9170c0fce475 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:51.543961 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1374/2757] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:04:51 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:04:51.549294 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ea67d0-56fb-4e84-bc5e-d56b073a7509 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:51.559691 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ea67d0-56fb-4e84-bc5e-d56b073a7509 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:51.564245 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94763ce0-a1d6-4695-855d-144767235d5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:51.564834 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94763ce0-a1d6-4695-855d-144767235d5e admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:51.565109 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94763ce0-a1d6-4695-855d-144767235d5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:51.565373 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94763ce0-a1d6-4695-855d-144767235d5e admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:51.566012 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94763ce0-a1d6-4695-855d-144767235d5e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:51.567633 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94763ce0-a1d6-4695-855d-144767235d5e admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:51.578611 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1384/2758] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:04:51 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:04:51.583464 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb65f1f-9f91-4955-99f6-9170c0fce475 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:51.594185 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb65f1f-9f91-4955-99f6-9170c0fce475 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:51.598858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51175b60-138e-46ca-964d-67bd88041db2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:51.599506 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51175b60-138e-46ca-964d-67bd88041db2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:51.599789 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51175b60-138e-46ca-964d-67bd88041db2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:51.600070 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51175b60-138e-46ca-964d-67bd88041db2 admin admin] PATH_INFO: `/v3/projects/739385b3a6c9407eaddde7acd5afb4bb/users/d2585a05a4774931b8ee4105ac061ab4/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:51.615591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51175b60-138e-46ca-964d-67bd88041db2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=739385b3a6c9407eaddde7acd5afb4bb, user_id=d2585a05a4774931b8ee4105ac061ab4, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:51.620235 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51175b60-138e-46ca-964d-67bd88041db2 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:51.633393 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1375/2759] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:04:51 2022] PUT /identity/v3/projects/739385b3a6c9407eaddde7acd5afb4bb/users/d2585a05a4774931b8ee4105ac061ab4/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:04:51.699021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18555aef-335d-4fe8-87f5-dff193efe590 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:51.699304 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18555aef-335d-4fe8-87f5-dff193efe590 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:51.699743 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18555aef-335d-4fe8-87f5-dff193efe590 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:51.728351 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-18555aef-335d-4fe8-87f5-dff193efe590 None None] MFA Rules not processed for user `d2585a05a4774931b8ee4105ac061ab4`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:04:51.729550 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-18555aef-335d-4fe8-87f5-dff193efe590 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:51.794321 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1385/2760] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:04:51 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:04:51.805320 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51175b60-138e-46ca-964d-67bd88041db2 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:04:51.807039 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-51175b60-138e-46ca-964d-67bd88041db2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:04:51.878587 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51175b60-138e-46ca-964d-67bd88041db2 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:04:51.882190 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7947d79-6a24-4df5-a070-e08d8f253631 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:04:51.882811 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7947d79-6a24-4df5-a070-e08d8f253631 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:04:51.883088 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7947d79-6a24-4df5-a070-e08d8f253631 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:04:51.883355 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7947d79-6a24-4df5-a070-e08d8f253631 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:04:51.891594 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7947d79-6a24-4df5-a070-e08d8f253631 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:04:51.893033 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7947d79-6a24-4df5-a070-e08d8f253631 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:04:51.902230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1376/2761] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:04:51 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:05:01.790749 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=64,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:05:01.898029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=86,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:05:09.647449 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18555aef-335d-4fe8-87f5-dff193efe590 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:05:09.707858 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18555aef-335d-4fe8-87f5-dff193efe590 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:05:09.712511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372b70f8-bf08-4cd0-bcdb-7a2cc9a68bd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:05:09.713131 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372b70f8-bf08-4cd0-bcdb-7a2cc9a68bd3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:09.713408 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372b70f8-bf08-4cd0-bcdb-7a2cc9a68bd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:09.713680 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372b70f8-bf08-4cd0-bcdb-7a2cc9a68bd3 admin admin] PATH_INFO: `/v3/users/d2585a05a4774931b8ee4105ac061ab4` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:09.714802 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-372b70f8-bf08-4cd0-bcdb-7a2cc9a68bd3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d2585a05a4774931b8ee4105ac061ab4)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:05:09.717202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-372b70f8-bf08-4cd0-bcdb-7a2cc9a68bd3 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:05:09.761035 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-372b70f8-bf08-4cd0-bcdb-7a2cc9a68bd3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd2585a05a4774931b8ee4105ac061ab4'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:05:09.767944 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-372b70f8-bf08-4cd0-bcdb-7a2cc9a68bd3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd2585a05a4774931b8ee4105ac061ab4'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:05:09.779925 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-372b70f8-bf08-4cd0-bcdb-7a2cc9a68bd3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd2585a05a4774931b8ee4105ac061ab4'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:05:09.780547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-372b70f8-bf08-4cd0-bcdb-7a2cc9a68bd3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd2585a05a4774931b8ee4105ac061ab4'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:05:09.787590 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-372b70f8-bf08-4cd0-bcdb-7a2cc9a68bd3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd2585a05a4774931b8ee4105ac061ab4'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:05:09.793466 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1386/2762] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:05:09 2022] DELETE /identity/v3/users/d2585a05a4774931b8ee4105ac061ab4 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:05:09.798666 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19eb1c5c-1c1f-4bde-97b2-c71286ddb79f None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:09.799155 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19eb1c5c-1c1f-4bde-97b2-c71286ddb79f None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:09.799267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19eb1c5c-1c1f-4bde-97b2-c71286ddb79f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:09.818242 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-19eb1c5c-1c1f-4bde-97b2-c71286ddb79f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:05:09.819469 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-19eb1c5c-1c1f-4bde-97b2-c71286ddb79f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:05:09.877638 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1377/2763] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:05:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:05:09.887396 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372b70f8-bf08-4cd0-bcdb-7a2cc9a68bd3 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:05:09.888810 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-372b70f8-bf08-4cd0-bcdb-7a2cc9a68bd3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:05:09.966336 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372b70f8-bf08-4cd0-bcdb-7a2cc9a68bd3 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:05:09.970403 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de06e7fe-c8bc-411a-861a-cc4438cf6930 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:05:09.971031 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de06e7fe-c8bc-411a-861a-cc4438cf6930 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:09.971331 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de06e7fe-c8bc-411a-861a-cc4438cf6930 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:09.971598 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de06e7fe-c8bc-411a-861a-cc4438cf6930 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:09.980433 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de06e7fe-c8bc-411a-861a-cc4438cf6930 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:05:09.981890 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de06e7fe-c8bc-411a-861a-cc4438cf6930 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:05:09.991191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1387/2764] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:05:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:05:10.257253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19eb1c5c-1c1f-4bde-97b2-c71286ddb79f None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:05:10.259069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-19eb1c5c-1c1f-4bde-97b2-c71286ddb79f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:05:10.272597 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19eb1c5c-1c1f-4bde-97b2-c71286ddb79f None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:05:10.277300 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ba1c94-b337-4115-9592-a2a5b14d7364 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:05:10.277940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ba1c94-b337-4115-9592-a2a5b14d7364 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:10.278220 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ba1c94-b337-4115-9592-a2a5b14d7364 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:10.278521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ba1c94-b337-4115-9592-a2a5b14d7364 admin admin] PATH_INFO: `/v3/projects/739385b3a6c9407eaddde7acd5afb4bb` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:10.280447 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50ba1c94-b337-4115-9592-a2a5b14d7364 admin admin] RBAC: Authorizing `identity:delete_project(project_id=739385b3a6c9407eaddde7acd5afb4bb)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:05:10.282370 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50ba1c94-b337-4115-9592-a2a5b14d7364 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:05:10.329629 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-50ba1c94-b337-4115-9592-a2a5b14d7364 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '739385b3a6c9407eaddde7acd5afb4bb', 'request_id': 'req-50ba1c94-b337-4115-9592-a2a5b14d7364'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:05:10.335089 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-50ba1c94-b337-4115-9592-a2a5b14d7364 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '739385b3a6c9407eaddde7acd5afb4bb', 'request_id': 'req-50ba1c94-b337-4115-9592-a2a5b14d7364'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:05:10.342774 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-50ba1c94-b337-4115-9592-a2a5b14d7364 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '739385b3a6c9407eaddde7acd5afb4bb', 'request_id': 'req-50ba1c94-b337-4115-9592-a2a5b14d7364'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:05:10.354527 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-50ba1c94-b337-4115-9592-a2a5b14d7364 admin admin] The token cache is being invalidate because project 739385b3a6c9407eaddde7acd5afb4bb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:05:10.354855 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-50ba1c94-b337-4115-9592-a2a5b14d7364 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:05:10.355446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-50ba1c94-b337-4115-9592-a2a5b14d7364 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:05:10.356960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1378/2765] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:05:10 2022] DELETE /identity/v3/projects/739385b3a6c9407eaddde7acd5afb4bb => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 01:05:10.361808 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437fdcef-ae3d-49d9-88ab-b5efdf91701f None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:10.362094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437fdcef-ae3d-49d9-88ab-b5efdf91701f None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:10.362402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437fdcef-ae3d-49d9-88ab-b5efdf91701f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:10.370043 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:05:10.384700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-437fdcef-ae3d-49d9-88ab-b5efdf91701f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:05:10.385892 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-437fdcef-ae3d-49d9-88ab-b5efdf91701f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:05:10.443941 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1388/2766] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:05:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:05:10.452488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2894ff2-3016-427b-9561-73690c5d670b None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:10.452766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2894ff2-3016-427b-9561-73690c5d670b None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:10.453034 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2894ff2-3016-427b-9561-73690c5d670b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:10.473609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-d2894ff2-3016-427b-9561-73690c5d670b None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:05:10.474842 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-d2894ff2-3016-427b-9561-73690c5d670b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:05:10.484627 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1379/2767] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:05:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:05:10.488922 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437fdcef-ae3d-49d9-88ab-b5efdf91701f None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:05:10.499194 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437fdcef-ae3d-49d9-88ab-b5efdf91701f None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:05:10.506695 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b10fb95-cf3f-412f-8397-349b103cb07d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:05:10.507298 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b10fb95-cf3f-412f-8397-349b103cb07d admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:10.507575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b10fb95-cf3f-412f-8397-349b103cb07d admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:10.507847 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b10fb95-cf3f-412f-8397-349b103cb07d admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:10.508490 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b10fb95-cf3f-412f-8397-349b103cb07d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:05:10.508796 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b10fb95-cf3f-412f-8397-349b103cb07d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:05:10.510395 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b10fb95-cf3f-412f-8397-349b103cb07d admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:05:10.523257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1389/2768] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:05:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 01:05:10.527474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2894ff2-3016-427b-9561-73690c5d670b None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:05:10.536898 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2894ff2-3016-427b-9561-73690c5d670b None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:05:10.541525 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af984ca1-f4b1-4746-9be1-6284cf99c579 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:05:10.542196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af984ca1-f4b1-4746-9be1-6284cf99c579 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:10.542496 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af984ca1-f4b1-4746-9be1-6284cf99c579 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:10.542768 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af984ca1-f4b1-4746-9be1-6284cf99c579 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:10.543487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af984ca1-f4b1-4746-9be1-6284cf99c579 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:05:10.545413 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af984ca1-f4b1-4746-9be1-6284cf99c579 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:05:10.573134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1380/2769] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:05:10 2022] POST /identity/v3/projects => generated 371 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:05:10.578176 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b10fb95-cf3f-412f-8397-349b103cb07d admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:05:10.588635 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b10fb95-cf3f-412f-8397-349b103cb07d admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:05:10.644055 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5351bd42-2a17-4933-8b7b-02ef46bc4754 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:05:10.644659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5351bd42-2a17-4933-8b7b-02ef46bc4754 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:10.644941 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5351bd42-2a17-4933-8b7b-02ef46bc4754 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:10.645214 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5351bd42-2a17-4933-8b7b-02ef46bc4754 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:10.645842 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5351bd42-2a17-4933-8b7b-02ef46bc4754 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:05:10.647786 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5351bd42-2a17-4933-8b7b-02ef46bc4754 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:05:10.669420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1390/2770] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:05:10 2022] POST /identity/v3/users => generated 286 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:05:10.673522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af984ca1-f4b1-4746-9be1-6284cf99c579 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:05:10.683589 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af984ca1-f4b1-4746-9be1-6284cf99c579 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:05:10.688775 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1513d59c-4602-4ce7-ad1a-656203fc4f12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:05:10.689362 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1513d59c-4602-4ce7-ad1a-656203fc4f12 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:10.689637 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1513d59c-4602-4ce7-ad1a-656203fc4f12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:10.689909 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1513d59c-4602-4ce7-ad1a-656203fc4f12 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:10.690581 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1513d59c-4602-4ce7-ad1a-656203fc4f12 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:05:10.692151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1513d59c-4602-4ce7-ad1a-656203fc4f12 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:05:10.702858 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1381/2771] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:05:10 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:05:10.706803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5351bd42-2a17-4933-8b7b-02ef46bc4754 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:05:10.716669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5351bd42-2a17-4933-8b7b-02ef46bc4754 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:05:10.722360 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1bb9b1-d980-4cd5-9a41-31839a27cb74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:05:10.722983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1bb9b1-d980-4cd5-9a41-31839a27cb74 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:10.723266 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1bb9b1-d980-4cd5-9a41-31839a27cb74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:10.723536 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1bb9b1-d980-4cd5-9a41-31839a27cb74 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:10.724178 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1bb9b1-d980-4cd5-9a41-31839a27cb74 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:05:10.725756 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1bb9b1-d980-4cd5-9a41-31839a27cb74 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:05:10.736844 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1391/2772] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:05:10 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:05:10.740884 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1513d59c-4602-4ce7-ad1a-656203fc4f12 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:05:10.751591 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1513d59c-4602-4ce7-ad1a-656203fc4f12 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:05:10.756195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f2296e-3197-4c82-a93c-13d4baea293b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:05:10.756862 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f2296e-3197-4c82-a93c-13d4baea293b admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:10.757141 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f2296e-3197-4c82-a93c-13d4baea293b admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:10.757409 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f2296e-3197-4c82-a93c-13d4baea293b admin admin] PATH_INFO: `/v3/projects/2d812ad5fce24a49b0f2d07081d5718a/users/7ea84c1e0dcf441fb1346eaaef4de421/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:10.771744 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f2296e-3197-4c82-a93c-13d4baea293b admin admin] RBAC: Authorizing `identity:create_grant(project_id=2d812ad5fce24a49b0f2d07081d5718a, user_id=7ea84c1e0dcf441fb1346eaaef4de421, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:05:10.774358 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f2296e-3197-4c82-a93c-13d4baea293b admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:05:10.787528 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1382/2773] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:05:10 2022] PUT /identity/v3/projects/2d812ad5fce24a49b0f2d07081d5718a/users/7ea84c1e0dcf441fb1346eaaef4de421/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:05:10.792728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4395f2f-80a9-4a45-868a-b0460cfdeb15 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:10.793013 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4395f2f-80a9-4a45-868a-b0460cfdeb15 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:10.793283 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4395f2f-80a9-4a45-868a-b0460cfdeb15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:10.812827 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-f4395f2f-80a9-4a45-868a-b0460cfdeb15 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:05:10.814025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-f4395f2f-80a9-4a45-868a-b0460cfdeb15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:05:10.871615 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1392/2774] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:05:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:05:10.881908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f2296e-3197-4c82-a93c-13d4baea293b admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:05:10.883715 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-a6f2296e-3197-4c82-a93c-13d4baea293b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:05:10.905553 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:05:10.942286 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f2296e-3197-4c82-a93c-13d4baea293b admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:05:10.945984 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b93c84b-2f71-4744-9d8f-f6dfc953cf2d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:05:10.946617 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b93c84b-2f71-4744-9d8f-f6dfc953cf2d service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:10.946893 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b93c84b-2f71-4744-9d8f-f6dfc953cf2d service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:10.947267 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b93c84b-2f71-4744-9d8f-f6dfc953cf2d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:10.956183 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b93c84b-2f71-4744-9d8f-f6dfc953cf2d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:05:10.957589 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b93c84b-2f71-4744-9d8f-f6dfc953cf2d service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:05:10.967068 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1383/2775] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:05:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:05:15.104974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c3172b-83d5-433c-8fd8-9c251373b82f None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:15.105462 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c3172b-83d5-433c-8fd8-9c251373b82f None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:15.105580 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c3172b-83d5-433c-8fd8-9c251373b82f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:15.126307 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:05:15.139754 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-52c3172b-83d5-433c-8fd8-9c251373b82f None None] MFA Rules not processed for user `7ea84c1e0dcf441fb1346eaaef4de421`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:05:15.140907 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-52c3172b-83d5-433c-8fd8-9c251373b82f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:05:15.205512 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1393/2776] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:05:15 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:05:15.209983 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b93c84b-2f71-4744-9d8f-f6dfc953cf2d service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:05:15.220010 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b93c84b-2f71-4744-9d8f-f6dfc953cf2d service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:05:15.224609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66af4f79-a112-4865-9bc3-b73b8352fb0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:05:15.225212 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66af4f79-a112-4865-9bc3-b73b8352fb0c admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:15.225495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66af4f79-a112-4865-9bc3-b73b8352fb0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:15.225766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66af4f79-a112-4865-9bc3-b73b8352fb0c admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:15.226397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66af4f79-a112-4865-9bc3-b73b8352fb0c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:05:15.228299 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66af4f79-a112-4865-9bc3-b73b8352fb0c admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:05:15.259395 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1384/2777] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:05:15 2022] POST /identity/v3/projects => generated 371 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 01:05:15.260745 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c3172b-83d5-433c-8fd8-9c251373b82f None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:05:15.271390 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c3172b-83d5-433c-8fd8-9c251373b82f None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:05:15.336388 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c2f068-ab13-4860-832c-3bea909ae479 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:05:15.337029 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c2f068-ab13-4860-832c-3bea909ae479 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:15.337308 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c2f068-ab13-4860-832c-3bea909ae479 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:15.337579 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c2f068-ab13-4860-832c-3bea909ae479 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:15.338243 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c2f068-ab13-4860-832c-3bea909ae479 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:05:15.340234 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c2f068-ab13-4860-832c-3bea909ae479 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:05:15.364322 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1394/2778] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:05:15 2022] POST /identity/v3/users => generated 292 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:05:15.368127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66af4f79-a112-4865-9bc3-b73b8352fb0c admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:05:15.379350 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66af4f79-a112-4865-9bc3-b73b8352fb0c admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:05:15.384192 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ec8c02-1b71-4487-9548-bb51e38cc8a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:05:15.384905 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ec8c02-1b71-4487-9548-bb51e38cc8a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:15.385264 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ec8c02-1b71-4487-9548-bb51e38cc8a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:15.385607 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ec8c02-1b71-4487-9548-bb51e38cc8a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:15.386257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ec8c02-1b71-4487-9548-bb51e38cc8a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:05:15.387913 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ec8c02-1b71-4487-9548-bb51e38cc8a9 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:05:15.399771 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1385/2779] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:05:15 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:05:15.404015 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c2f068-ab13-4860-832c-3bea909ae479 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:05:15.414191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c2f068-ab13-4860-832c-3bea909ae479 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:05:15.419154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e41e72-c044-45f3-ab20-97775e08561c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:05:15.420306 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e41e72-c044-45f3-ab20-97775e08561c admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:15.420584 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e41e72-c044-45f3-ab20-97775e08561c admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:15.420856 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e41e72-c044-45f3-ab20-97775e08561c admin admin] PATH_INFO: `/v3/projects/ba1f93ecb12540f5bb7f7e0e958776e5/users/b19a882b58b14abda11e33e416356b99/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:15.437623 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e41e72-c044-45f3-ab20-97775e08561c admin admin] RBAC: Authorizing `identity:create_grant(project_id=ba1f93ecb12540f5bb7f7e0e958776e5, user_id=b19a882b58b14abda11e33e416356b99, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:05:15.440323 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e41e72-c044-45f3-ab20-97775e08561c admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:05:15.454831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1395/2780] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:05:15 2022] PUT /identity/v3/projects/ba1f93ecb12540f5bb7f7e0e958776e5/users/b19a882b58b14abda11e33e416356b99/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:05:15.463032 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ec8c02-1b71-4487-9548-bb51e38cc8a9 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:05:15.473442 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ec8c02-1b71-4487-9548-bb51e38cc8a9 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:05:15.564745 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc97755-173c-44dc-b2c7-8aab81555a4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:05:15.565614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc97755-173c-44dc-b2c7-8aab81555a4d admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:15.566008 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc97755-173c-44dc-b2c7-8aab81555a4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:15.566377 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc97755-173c-44dc-b2c7-8aab81555a4d admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:15.567160 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc97755-173c-44dc-b2c7-8aab81555a4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:05:15.568741 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc97755-173c-44dc-b2c7-8aab81555a4d admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:05:15.579557 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1386/2781] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:05:15 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:05:15.583912 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e41e72-c044-45f3-ab20-97775e08561c admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:05:15.595252 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e41e72-c044-45f3-ab20-97775e08561c admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:05:15.600161 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89f783d-7d72-4327-bc3e-cb5ddcedb05b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:05:15.600836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89f783d-7d72-4327-bc3e-cb5ddcedb05b admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:15.601493 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89f783d-7d72-4327-bc3e-cb5ddcedb05b admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:15.601773 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89f783d-7d72-4327-bc3e-cb5ddcedb05b admin admin] PATH_INFO: `/v3/projects/ba1f93ecb12540f5bb7f7e0e958776e5/users/b19a882b58b14abda11e33e416356b99/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:15.603734 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89f783d-7d72-4327-bc3e-cb5ddcedb05b admin admin] RBAC: Authorizing `identity:create_grant(project_id=ba1f93ecb12540f5bb7f7e0e958776e5, user_id=b19a882b58b14abda11e33e416356b99, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:05:15.606422 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89f783d-7d72-4327-bc3e-cb5ddcedb05b admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:05:15.618150 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1396/2782] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:05:15 2022] PUT /identity/v3/projects/ba1f93ecb12540f5bb7f7e0e958776e5/users/b19a882b58b14abda11e33e416356b99/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:05:19.868093 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f5c70c-128b-446b-9605-a814583ad79a None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:19.868949 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f5c70c-128b-446b-9605-a814583ad79a None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:19.869225 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f5c70c-128b-446b-9605-a814583ad79a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:19.897723 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-84f5c70c-128b-446b-9605-a814583ad79a None None] MFA Rules not processed for user `b19a882b58b14abda11e33e416356b99`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:05:19.899024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-84f5c70c-128b-446b-9605-a814583ad79a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:05:19.971712 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1387/2783] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:05:19 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:05:19.987816 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89f783d-7d72-4327-bc3e-cb5ddcedb05b admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:05:19.989533 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-b89f783d-7d72-4327-bc3e-cb5ddcedb05b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:05:20.052622 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:05:20.103917 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89f783d-7d72-4327-bc3e-cb5ddcedb05b admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:05:20.109621 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95634bf8-fbfe-4651-8d62-ece2e53838f5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:05:20.110232 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95634bf8-fbfe-4651-8d62-ece2e53838f5 service nova] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:20.110543 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95634bf8-fbfe-4651-8d62-ece2e53838f5 service nova] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:20.110821 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95634bf8-fbfe-4651-8d62-ece2e53838f5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:20.167817 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95634bf8-fbfe-4651-8d62-ece2e53838f5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:05:20.169399 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95634bf8-fbfe-4651-8d62-ece2e53838f5 service nova] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:05:20.178160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1397/2784] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:05:19 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:05:21.221321 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f163ad2-6737-406d-af62-de1054ab0da0 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:21.221840 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f163ad2-6737-406d-af62-de1054ab0da0 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:21.221840 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f163ad2-6737-406d-af62-de1054ab0da0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:21.240759 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-3f163ad2-6737-406d-af62-de1054ab0da0 None None] MFA Rules not processed for user `7ea84c1e0dcf441fb1346eaaef4de421`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:05:21.241957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-3f163ad2-6737-406d-af62-de1054ab0da0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:05:21.248812 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1388/2785] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:05:21 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:05:30.174052 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=45 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:05:31.235324 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:05:36.542722 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95634bf8-fbfe-4651-8d62-ece2e53838f5 service nova] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:05:36.544129 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-95634bf8-fbfe-4651-8d62-ece2e53838f5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:05:36.620121 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95634bf8-fbfe-4651-8d62-ece2e53838f5 service nova] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:05:36.623618 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625bd36b-2f24-4ea0-9779-4b17ad5c6982 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:05:36.624223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625bd36b-2f24-4ea0-9779-4b17ad5c6982 service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:36.624501 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625bd36b-2f24-4ea0-9779-4b17ad5c6982 service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:36.624770 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625bd36b-2f24-4ea0-9779-4b17ad5c6982 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:36.627084 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-625bd36b-2f24-4ea0-9779-4b17ad5c6982 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:05:36.683147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625bd36b-2f24-4ea0-9779-4b17ad5c6982 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:05:36.684583 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625bd36b-2f24-4ea0-9779-4b17ad5c6982 service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:05:36.693913 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1398/2786] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:05:36 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:05:39.944509 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f163ad2-6737-406d-af62-de1054ab0da0 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:05:39.946053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-3f163ad2-6737-406d-af62-de1054ab0da0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:05:40.037985 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f163ad2-6737-406d-af62-de1054ab0da0 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:05:40.043157 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7c8c2a-f628-42b9-a20f-9828b41cdcff service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8f30116fb5ce43dab69efa8bafd9d049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:05:40.043804 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7c8c2a-f628-42b9-a20f-9828b41cdcff service placement] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:40.044037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7c8c2a-f628-42b9-a20f-9828b41cdcff service placement] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:40.044347 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7c8c2a-f628-42b9-a20f-9828b41cdcff service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:40.046354 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-6d7c8c2a-f628-42b9-a20f-9828b41cdcff service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:05:40.058613 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7c8c2a-f628-42b9-a20f-9828b41cdcff service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:05:40.060322 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7c8c2a-f628-42b9-a20f-9828b41cdcff service placement] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:05:40.075664 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1389/2787] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:05:39 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:05:44.631171 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079376a7-9fe7-4cef-85c4-a398db7ca9fe None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:44.631658 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079376a7-9fe7-4cef-85c4-a398db7ca9fe None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:44.631783 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079376a7-9fe7-4cef-85c4-a398db7ca9fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:44.650331 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-079376a7-9fe7-4cef-85c4-a398db7ca9fe None None] MFA Rules not processed for user `823fa0c3eeeb489591c639b81afc02bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:05:44.651545 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-079376a7-9fe7-4cef-85c4-a398db7ca9fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:05:44.658186 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1399/2788] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:05:44 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:05:50.067562 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=41 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:05:54.644860 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=60 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:05:58.220647 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7c8c2a-f628-42b9-a20f-9828b41cdcff service placement] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:05:58.223193 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-6d7c8c2a-f628-42b9-a20f-9828b41cdcff service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:05:58.282878 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7c8c2a-f628-42b9-a20f-9828b41cdcff service placement] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:05:58.286562 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a451d2-265c-4ed3-80aa-e34cf0c5db2e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:05:58.287163 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a451d2-265c-4ed3-80aa-e34cf0c5db2e service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:05:58.287441 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a451d2-265c-4ed3-80aa-e34cf0c5db2e service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:05:58.287707 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a451d2-265c-4ed3-80aa-e34cf0c5db2e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:05:58.296477 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a451d2-265c-4ed3-80aa-e34cf0c5db2e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:05:58.297889 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a451d2-265c-4ed3-80aa-e34cf0c5db2e service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:05:58.307650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1390/2789] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:05:58 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:06:08.303707 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=26 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:06:50.692695 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079376a7-9fe7-4cef-85c4-a398db7ca9fe None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:06:50.703191 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079376a7-9fe7-4cef-85c4-a398db7ca9fe None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:06:50.706850 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8456c0-ea73-4e77-8400-266955ce023c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:06:50.707476 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8456c0-ea73-4e77-8400-266955ce023c service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:06:50.707752 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8456c0-ea73-4e77-8400-266955ce023c service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:06:50.708017 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8456c0-ea73-4e77-8400-266955ce023c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:06:50.710343 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-7a8456c0-ea73-4e77-8400-266955ce023c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:06:50.724769 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8456c0-ea73-4e77-8400-266955ce023c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:06:50.726197 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8456c0-ea73-4e77-8400-266955ce023c service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:06:50.737222 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1400/2790] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:06:50 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:07:00.732411 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=6 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:07:26.528050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a451d2-265c-4ed3-80aa-e34cf0c5db2e service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:07:26.536776 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a451d2-265c-4ed3-80aa-e34cf0c5db2e service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:07:26.540324 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7322180e-908e-402d-b891-3e72d2c9b262 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:07:26.540908 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7322180e-908e-402d-b891-3e72d2c9b262 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:07:26.541182 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7322180e-908e-402d-b891-3e72d2c9b262 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:07:26.541449 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7322180e-908e-402d-b891-3e72d2c9b262 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:07:26.543151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-7322180e-908e-402d-b891-3e72d2c9b262 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:07:26.556754 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7322180e-908e-402d-b891-3e72d2c9b262 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:07:26.558155 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7322180e-908e-402d-b891-3e72d2c9b262 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:07:26.569899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1391/2791] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:07:26 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:07:33.130363 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1036126-a89b-421d-85b6-b22a8fd84e94 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:07:33.130928 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1036126-a89b-421d-85b6-b22a8fd84e94 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:07:33.130928 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1036126-a89b-421d-85b6-b22a8fd84e94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:07:33.150198 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-a1036126-a89b-421d-85b6-b22a8fd84e94 None None] MFA Rules not processed for user `bd2fa2deb0a24aff9d8b7ce97b6e461b`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:07:33.152565 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a1036126-a89b-421d-85b6-b22a8fd84e94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:07:33.213782 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1401/2792] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:07:33 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:07:36.564112 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=6 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:07:43.206732 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=25 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:07:45.653468 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7322180e-908e-402d-b891-3e72d2c9b262 service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:07:45.655217 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-7322180e-908e-402d-b891-3e72d2c9b262 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:07:45.669189 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7322180e-908e-402d-b891-3e72d2c9b262 service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:07:45.675856 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c57d833-3aef-4d08-bbeb-561d649924e5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:07:45.676629 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c57d833-3aef-4d08-bbeb-561d649924e5 service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:07:45.677012 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c57d833-3aef-4d08-bbeb-561d649924e5 service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:07:45.677469 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c57d833-3aef-4d08-bbeb-561d649924e5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:07:45.679373 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-4c57d833-3aef-4d08-bbeb-561d649924e5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:07:45.692700 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c57d833-3aef-4d08-bbeb-561d649924e5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:07:45.695134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c57d833-3aef-4d08-bbeb-561d649924e5 service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:07:45.706993 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1392/2793] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:07:45 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:07:45.801319 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1036126-a89b-421d-85b6-b22a8fd84e94 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:07:45.819633 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c57d833-3aef-4d08-bbeb-561d649924e5 service nova] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:07:45.826094 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1036126-a89b-421d-85b6-b22a8fd84e94 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:07:45.831393 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c57d833-3aef-4d08-bbeb-561d649924e5 service nova] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:07:45.836511 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e2cd76-b21a-4467-8a11-e30f3b96c81d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:07:45.838576 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e2cd76-b21a-4467-8a11-e30f3b96c81d service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:07:45.838974 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04181a49-8420-4329-8572-abf08c3cda4c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:07:45.839387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e2cd76-b21a-4467-8a11-e30f3b96c81d service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:07:45.840387 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04181a49-8420-4329-8572-abf08c3cda4c service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:07:45.840754 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e2cd76-b21a-4467-8a11-e30f3b96c81d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:07:45.841015 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04181a49-8420-4329-8572-abf08c3cda4c service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:07:45.841446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04181a49-8420-4329-8572-abf08c3cda4c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:07:45.843775 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-04181a49-8420-4329-8572-abf08c3cda4c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:07:45.846692 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a6e2cd76-b21a-4467-8a11-e30f3b96c81d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:07:45.858299 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04181a49-8420-4329-8572-abf08c3cda4c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:07:45.860196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04181a49-8420-4329-8572-abf08c3cda4c service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:07:45.870494 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e2cd76-b21a-4467-8a11-e30f3b96c81d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:07:45.876309 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e2cd76-b21a-4467-8a11-e30f3b96c81d service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:07:45.885648 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1393/2794] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:07:45 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:07:45.897518 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1402/2795] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:07:45 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:07:55.867836 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=12 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:07:55.890023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=6 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:08:00.617896 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04181a49-8420-4329-8572-abf08c3cda4c service nova] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:08:00.628126 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04181a49-8420-4329-8572-abf08c3cda4c service nova] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:08:00.631745 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5fd78e-25f5-4c3d-b6bd-993400943e6c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:08:00.632338 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5fd78e-25f5-4c3d-b6bd-993400943e6c service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:00.633105 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5fd78e-25f5-4c3d-b6bd-993400943e6c service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:00.633376 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5fd78e-25f5-4c3d-b6bd-993400943e6c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:00.635138 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8a5fd78e-25f5-4c3d-b6bd-993400943e6c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:08:00.700131 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5fd78e-25f5-4c3d-b6bd-993400943e6c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:08:00.701590 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5fd78e-25f5-4c3d-b6bd-993400943e6c service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:08:00.711645 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1394/2796] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:08:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:08:07.384542 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e2cd76-b21a-4467-8a11-e30f3b96c81d service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:08:07.386194 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-a6e2cd76-b21a-4467-8a11-e30f3b96c81d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:08:07.401625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e2cd76-b21a-4467-8a11-e30f3b96c81d service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:08:07.406201 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826ba101-0d18-4c21-8185-baaf033af7a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:08:07.406874 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826ba101-0d18-4c21-8185-baaf033af7a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:07.407151 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826ba101-0d18-4c21-8185-baaf033af7a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:07.407421 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826ba101-0d18-4c21-8185-baaf033af7a7 admin admin] PATH_INFO: `/v3/users/823fa0c3eeeb489591c639b81afc02bc` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:07.408551 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-826ba101-0d18-4c21-8185-baaf033af7a7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=823fa0c3eeeb489591c639b81afc02bc)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:08:07.410487 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-826ba101-0d18-4c21-8185-baaf033af7a7 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:08:07.458810 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-826ba101-0d18-4c21-8185-baaf033af7a7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '823fa0c3eeeb489591c639b81afc02bc'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:07.467074 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-826ba101-0d18-4c21-8185-baaf033af7a7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '823fa0c3eeeb489591c639b81afc02bc'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:07.480953 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-826ba101-0d18-4c21-8185-baaf033af7a7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '823fa0c3eeeb489591c639b81afc02bc'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:07.481699 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-826ba101-0d18-4c21-8185-baaf033af7a7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '823fa0c3eeeb489591c639b81afc02bc'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:07.489975 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-826ba101-0d18-4c21-8185-baaf033af7a7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '823fa0c3eeeb489591c639b81afc02bc'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:07.496761 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1403/2797] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:08:07 2022] DELETE /identity/v3/users/823fa0c3eeeb489591c639b81afc02bc => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:08:07.726987 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5fd78e-25f5-4c3d-b6bd-993400943e6c service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:08:07.737379 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5fd78e-25f5-4c3d-b6bd-993400943e6c service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:08:07.802025 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226e2f38-3ae0-47b6-a1f5-66f84b411855 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:08:07.802677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226e2f38-3ae0-47b6-a1f5-66f84b411855 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:07.802955 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226e2f38-3ae0-47b6-a1f5-66f84b411855 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:07.803224 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226e2f38-3ae0-47b6-a1f5-66f84b411855 admin admin] PATH_INFO: `/v3/projects/6958209222b44905bcb4baa1b8c26c58` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:07.804877 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226e2f38-3ae0-47b6-a1f5-66f84b411855 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6958209222b44905bcb4baa1b8c26c58)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:08:07.806806 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226e2f38-3ae0-47b6-a1f5-66f84b411855 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:08:07.858857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-226e2f38-3ae0-47b6-a1f5-66f84b411855 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6958209222b44905bcb4baa1b8c26c58', 'request_id': 'req-226e2f38-3ae0-47b6-a1f5-66f84b411855'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:07.867490 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-226e2f38-3ae0-47b6-a1f5-66f84b411855 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6958209222b44905bcb4baa1b8c26c58', 'request_id': 'req-226e2f38-3ae0-47b6-a1f5-66f84b411855'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:07.877403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-226e2f38-3ae0-47b6-a1f5-66f84b411855 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6958209222b44905bcb4baa1b8c26c58', 'request_id': 'req-226e2f38-3ae0-47b6-a1f5-66f84b411855'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:07.891012 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-226e2f38-3ae0-47b6-a1f5-66f84b411855 admin admin] The token cache is being invalidate because project 6958209222b44905bcb4baa1b8c26c58 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:08:07.891536 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-226e2f38-3ae0-47b6-a1f5-66f84b411855 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:07.892590 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-226e2f38-3ae0-47b6-a1f5-66f84b411855 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:07.894201 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1395/2798] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:08:07 2022] DELETE /identity/v3/projects/6958209222b44905bcb4baa1b8c26c58 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:08:07.898300 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826ba101-0d18-4c21-8185-baaf033af7a7 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:08:07.899768 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-826ba101-0d18-4c21-8185-baaf033af7a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:08:07.912685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826ba101-0d18-4c21-8185-baaf033af7a7 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:08:07.967862 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c797a045-341d-445a-98cf-a4e733fd3432 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:08:07.968494 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c797a045-341d-445a-98cf-a4e733fd3432 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:07.968774 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c797a045-341d-445a-98cf-a4e733fd3432 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:07.969047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c797a045-341d-445a-98cf-a4e733fd3432 admin admin] PATH_INFO: `/v3/users/d7ae4c525e6848b48d48d554bf36a644` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:07.970365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c797a045-341d-445a-98cf-a4e733fd3432 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d7ae4c525e6848b48d48d554bf36a644)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:08:07.972343 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c797a045-341d-445a-98cf-a4e733fd3432 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:08:08.016777 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-c797a045-341d-445a-98cf-a4e733fd3432 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd7ae4c525e6848b48d48d554bf36a644'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:08.024063 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-c797a045-341d-445a-98cf-a4e733fd3432 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd7ae4c525e6848b48d48d554bf36a644'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:08.035835 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-c797a045-341d-445a-98cf-a4e733fd3432 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd7ae4c525e6848b48d48d554bf36a644'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:08.036477 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-c797a045-341d-445a-98cf-a4e733fd3432 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd7ae4c525e6848b48d48d554bf36a644'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:08.038420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:08:08.046065 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-c797a045-341d-445a-98cf-a4e733fd3432 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd7ae4c525e6848b48d48d554bf36a644'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:08.052299 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1404/2799] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:08:07 2022] DELETE /identity/v3/users/d7ae4c525e6848b48d48d554bf36a644 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:08:08.224430 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226e2f38-3ae0-47b6-a1f5-66f84b411855 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:08:08.225869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-226e2f38-3ae0-47b6-a1f5-66f84b411855 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:08:08.240106 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226e2f38-3ae0-47b6-a1f5-66f84b411855 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:08:08.260469 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:08:08.299578 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b508404d-a14f-4e4a-9af0-def5733dd5f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:08:08.300133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b508404d-a14f-4e4a-9af0-def5733dd5f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:08.300414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b508404d-a14f-4e4a-9af0-def5733dd5f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:08.300689 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b508404d-a14f-4e4a-9af0-def5733dd5f2 admin admin] PATH_INFO: `/v3/projects/bf2fa0f0609f48d2988f950b3878b5c0` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:08.301801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b508404d-a14f-4e4a-9af0-def5733dd5f2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bf2fa0f0609f48d2988f950b3878b5c0)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:08:08.303742 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b508404d-a14f-4e4a-9af0-def5733dd5f2 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:08:08.350299 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b508404d-a14f-4e4a-9af0-def5733dd5f2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bf2fa0f0609f48d2988f950b3878b5c0', 'request_id': 'req-b508404d-a14f-4e4a-9af0-def5733dd5f2'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:08.356592 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b508404d-a14f-4e4a-9af0-def5733dd5f2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bf2fa0f0609f48d2988f950b3878b5c0', 'request_id': 'req-b508404d-a14f-4e4a-9af0-def5733dd5f2'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:08.364128 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b508404d-a14f-4e4a-9af0-def5733dd5f2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bf2fa0f0609f48d2988f950b3878b5c0', 'request_id': 'req-b508404d-a14f-4e4a-9af0-def5733dd5f2'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:08.375357 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b508404d-a14f-4e4a-9af0-def5733dd5f2 admin admin] The token cache is being invalidate because project bf2fa0f0609f48d2988f950b3878b5c0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:08:08.375642 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b508404d-a14f-4e4a-9af0-def5733dd5f2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:08.376272 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-b508404d-a14f-4e4a-9af0-def5733dd5f2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:08.377747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1396/2800] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:08:08 2022] DELETE /identity/v3/projects/bf2fa0f0609f48d2988f950b3878b5c0 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:08:12.550360 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da38a366-a833-4768-8963-e476c20ecc00 None None] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:12.550960 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da38a366-a833-4768-8963-e476c20ecc00 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:12.550960 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da38a366-a833-4768-8963-e476c20ecc00 None None] PATH_INFO: `/` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:12.552309 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1405/2801] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 18 01:08:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 01:08:12.558730 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03318c72-b808-4a95-a457-afb7e717d6a3 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:12.559221 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03318c72-b808-4a95-a457-afb7e717d6a3 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:12.559422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03318c72-b808-4a95-a457-afb7e717d6a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:12.609329 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-03318c72-b808-4a95-a457-afb7e717d6a3 None None] MFA Rules not processed for user `9b0510812b60454586a781e6399e77a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:08:12.610628 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-03318c72-b808-4a95-a457-afb7e717d6a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:08:12.661931 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1397/2802] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 18 01:08:12 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:08:12.675003 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da38a366-a833-4768-8963-e476c20ecc00 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:08:12.676685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-da38a366-a833-4768-8963-e476c20ecc00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:08:12.690002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da38a366-a833-4768-8963-e476c20ecc00 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:08:12.693467 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51441985-ee3e-4b29-b557-a92ab2481502 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:08:12.694056 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51441985-ee3e-4b29-b557-a92ab2481502 service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:12.694436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51441985-ee3e-4b29-b557-a92ab2481502 service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:12.694797 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51441985-ee3e-4b29-b557-a92ab2481502 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:12.702540 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51441985-ee3e-4b29-b557-a92ab2481502 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:08:12.703961 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51441985-ee3e-4b29-b557-a92ab2481502 service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:08:12.712091 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1406/2803] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:08:12 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:08:13.802298 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47181ae-f1b9-43ce-a799-db281602b569 None None] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:13.802862 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47181ae-f1b9-43ce-a799-db281602b569 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:13.803017 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47181ae-f1b9-43ce-a799-db281602b569 None None] PATH_INFO: `/` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:13.804432 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1398/2804] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 18 01:08:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 18 01:08:13.810736 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bea9661-7c8d-49fa-9cd0-c75cc1c38ac7 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:13.811203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bea9661-7c8d-49fa-9cd0-c75cc1c38ac7 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:13.811425 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bea9661-7c8d-49fa-9cd0-c75cc1c38ac7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:13.849319 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-1bea9661-7c8d-49fa-9cd0-c75cc1c38ac7 None None] MFA Rules not processed for user `a59d73fd45a54651930d52300704fdb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:08:13.850537 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-1bea9661-7c8d-49fa-9cd0-c75cc1c38ac7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:08:13.916281 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1407/2805] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 18 01:08:13 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 01:08:13.926800 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47181ae-f1b9-43ce-a799-db281602b569 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:08:13.938326 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47181ae-f1b9-43ce-a799-db281602b569 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:08:13.941839 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773801b3-c60b-4f59-b7df-5e72ea3e4b85 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:08:13.942427 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773801b3-c60b-4f59-b7df-5e72ea3e4b85 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:13.942745 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773801b3-c60b-4f59-b7df-5e72ea3e4b85 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:13.943014 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773801b3-c60b-4f59-b7df-5e72ea3e4b85 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:13.952647 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773801b3-c60b-4f59-b7df-5e72ea3e4b85 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:08:13.954073 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773801b3-c60b-4f59-b7df-5e72ea3e4b85 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:08:13.964473 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1399/2806] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:08:13 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 18 01:08:14.029933 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bea9661-7c8d-49fa-9cd0-c75cc1c38ac7 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:08:14.031405 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-1bea9661-7c8d-49fa-9cd0-c75cc1c38ac7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:08:14.100474 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bea9661-7c8d-49fa-9cd0-c75cc1c38ac7 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:08:14.106105 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18cd080-2fc9-4f15-b99c-0648f2f956bd service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8f30116fb5ce43dab69efa8bafd9d049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:08:14.106743 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18cd080-2fc9-4f15-b99c-0648f2f956bd service placement] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:14.107021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18cd080-2fc9-4f15-b99c-0648f2f956bd service placement] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:14.107295 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18cd080-2fc9-4f15-b99c-0648f2f956bd service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:14.109014 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-e18cd080-2fc9-4f15-b99c-0648f2f956bd service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:08:14.120802 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18cd080-2fc9-4f15-b99c-0648f2f956bd service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:08:14.122368 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18cd080-2fc9-4f15-b99c-0648f2f956bd service placement] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:08:14.132090 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1408/2807] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:08:14 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:08:16.042776 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773801b3-c60b-4f59-b7df-5e72ea3e4b85 service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:08:16.044228 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-773801b3-c60b-4f59-b7df-5e72ea3e4b85 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:08:16.062315 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773801b3-c60b-4f59-b7df-5e72ea3e4b85 service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:08:16.068325 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568d386c-d296-443f-9be5-2c70c1ff1c2d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a59d73fd45a54651930d52300704fdb0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:08:16.068959 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568d386c-d296-443f-9be5-2c70c1ff1c2d service nova] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:16.069237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568d386c-d296-443f-9be5-2c70c1ff1c2d service nova] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:16.069515 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568d386c-d296-443f-9be5-2c70c1ff1c2d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:16.072365 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-568d386c-d296-443f-9be5-2c70c1ff1c2d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:08:16.087713 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568d386c-d296-443f-9be5-2c70c1ff1c2d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:08:16.089536 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568d386c-d296-443f-9be5-2c70c1ff1c2d service nova] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:08:16.112251 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1400/2808] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:08:16 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:08:24.127856 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=79 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:08:26.112468 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=85,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:08:56.372461 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18cd080-2fc9-4f15-b99c-0648f2f956bd service placement] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:08:56.373964 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-e18cd080-2fc9-4f15-b99c-0648f2f956bd service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:08:56.446635 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18cd080-2fc9-4f15-b99c-0648f2f956bd service placement] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:08:56.451753 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb47926c-4bc2-4fda-8bc5-72ca21e3e3a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:08:56.452783 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47926c-4bc2-4fda-8bc5-72ca21e3e3a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:56.453063 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47926c-4bc2-4fda-8bc5-72ca21e3e3a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:56.453370 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47926c-4bc2-4fda-8bc5-72ca21e3e3a5 admin admin] PATH_INFO: `/v3/users/bd2fa2deb0a24aff9d8b7ce97b6e461b` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:56.459443 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb47926c-4bc2-4fda-8bc5-72ca21e3e3a5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bd2fa2deb0a24aff9d8b7ce97b6e461b)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:08:56.461774 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb47926c-4bc2-4fda-8bc5-72ca21e3e3a5 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:08:56.526053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-bb47926c-4bc2-4fda-8bc5-72ca21e3e3a5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bd2fa2deb0a24aff9d8b7ce97b6e461b'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:56.536835 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-bb47926c-4bc2-4fda-8bc5-72ca21e3e3a5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bd2fa2deb0a24aff9d8b7ce97b6e461b'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:56.551685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-bb47926c-4bc2-4fda-8bc5-72ca21e3e3a5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bd2fa2deb0a24aff9d8b7ce97b6e461b'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:56.552639 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-bb47926c-4bc2-4fda-8bc5-72ca21e3e3a5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bd2fa2deb0a24aff9d8b7ce97b6e461b'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:56.559650 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-bb47926c-4bc2-4fda-8bc5-72ca21e3e3a5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bd2fa2deb0a24aff9d8b7ce97b6e461b'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:56.569086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1409/2809] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:08:56 2022] DELETE /identity/v3/users/bd2fa2deb0a24aff9d8b7ce97b6e461b => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:08:56.744017 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568d386c-d296-443f-9be5-2c70c1ff1c2d service nova] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:08:56.754929 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568d386c-d296-443f-9be5-2c70c1ff1c2d service nova] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:08:56.811427 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe94cf4b-0b14-461b-8a43-7b0bc00253d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:08:56.812047 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe94cf4b-0b14-461b-8a43-7b0bc00253d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:56.812322 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe94cf4b-0b14-461b-8a43-7b0bc00253d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:56.812586 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe94cf4b-0b14-461b-8a43-7b0bc00253d3 admin admin] PATH_INFO: `/v3/projects/39509a7e0ca64f94913b5fbf10c6ec9c` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:56.813687 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe94cf4b-0b14-461b-8a43-7b0bc00253d3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=39509a7e0ca64f94913b5fbf10c6ec9c)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:08:56.815615 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe94cf4b-0b14-461b-8a43-7b0bc00253d3 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:08:56.861153 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-fe94cf4b-0b14-461b-8a43-7b0bc00253d3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '39509a7e0ca64f94913b5fbf10c6ec9c', 'request_id': 'req-fe94cf4b-0b14-461b-8a43-7b0bc00253d3'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:56.866711 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-fe94cf4b-0b14-461b-8a43-7b0bc00253d3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '39509a7e0ca64f94913b5fbf10c6ec9c', 'request_id': 'req-fe94cf4b-0b14-461b-8a43-7b0bc00253d3'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:56.874064 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-fe94cf4b-0b14-461b-8a43-7b0bc00253d3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '39509a7e0ca64f94913b5fbf10c6ec9c', 'request_id': 'req-fe94cf4b-0b14-461b-8a43-7b0bc00253d3'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:56.885971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-fe94cf4b-0b14-461b-8a43-7b0bc00253d3 admin admin] The token cache is being invalidate because project 39509a7e0ca64f94913b5fbf10c6ec9c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:08:56.886262 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-fe94cf4b-0b14-461b-8a43-7b0bc00253d3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:56.886925 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-fe94cf4b-0b14-461b-8a43-7b0bc00253d3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:08:56.888388 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1401/2810] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:08:56 2022] DELETE /identity/v3/projects/39509a7e0ca64f94913b5fbf10c6ec9c => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:08:56.893377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2159b6a4-e1c0-4284-a6b5-d98463ebc23f None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:56.893666 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2159b6a4-e1c0-4284-a6b5-d98463ebc23f None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:56.893940 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2159b6a4-e1c0-4284-a6b5-d98463ebc23f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:56.913681 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-2159b6a4-e1c0-4284-a6b5-d98463ebc23f None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:08:56.914914 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-2159b6a4-e1c0-4284-a6b5-d98463ebc23f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:08:56.975977 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1410/2811] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:08:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:08:56.984141 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079827cf-d0b4-4eea-9591-8b20485ad331 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:56.984405 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079827cf-d0b4-4eea-9591-8b20485ad331 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:56.984675 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079827cf-d0b4-4eea-9591-8b20485ad331 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:57.004348 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-079827cf-d0b4-4eea-9591-8b20485ad331 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:08:57.005537 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-079827cf-d0b4-4eea-9591-8b20485ad331 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:08:57.014314 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1402/2812] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:08:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:08:57.018536 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2159b6a4-e1c0-4284-a6b5-d98463ebc23f None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:08:57.028696 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2159b6a4-e1c0-4284-a6b5-d98463ebc23f None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:08:57.033300 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bef8c5-2a25-4bbc-ad9b-ce679e87b2a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:08:57.033903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bef8c5-2a25-4bbc-ad9b-ce679e87b2a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:57.034184 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bef8c5-2a25-4bbc-ad9b-ce679e87b2a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:57.034473 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bef8c5-2a25-4bbc-ad9b-ce679e87b2a3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:57.035154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bef8c5-2a25-4bbc-ad9b-ce679e87b2a3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:08:57.035456 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bef8c5-2a25-4bbc-ad9b-ce679e87b2a3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:08:57.037084 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bef8c5-2a25-4bbc-ad9b-ce679e87b2a3 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:08:57.048814 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1411/2813] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:08:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 01:08:57.052941 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079827cf-d0b4-4eea-9591-8b20485ad331 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:08:57.062673 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079827cf-d0b4-4eea-9591-8b20485ad331 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:08:57.067435 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7687e8dd-ed54-4449-8140-f3805e1efb44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:08:57.068031 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7687e8dd-ed54-4449-8140-f3805e1efb44 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:57.068311 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7687e8dd-ed54-4449-8140-f3805e1efb44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:57.068583 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7687e8dd-ed54-4449-8140-f3805e1efb44 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:57.069217 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7687e8dd-ed54-4449-8140-f3805e1efb44 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:08:57.071146 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7687e8dd-ed54-4449-8140-f3805e1efb44 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:08:57.107781 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1403/2814] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:08:57 2022] POST /identity/v3/projects => generated 367 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:08:57.112001 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bef8c5-2a25-4bbc-ad9b-ce679e87b2a3 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:08:57.122376 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bef8c5-2a25-4bbc-ad9b-ce679e87b2a3 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:08:57.126278 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:08:57.179227 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561b608c-6a75-4d21-adb9-5988c315e73f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:08:57.179833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561b608c-6a75-4d21-adb9-5988c315e73f admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:57.180139 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561b608c-6a75-4d21-adb9-5988c315e73f admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:57.180413 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561b608c-6a75-4d21-adb9-5988c315e73f admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:57.181563 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561b608c-6a75-4d21-adb9-5988c315e73f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:08:57.183581 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561b608c-6a75-4d21-adb9-5988c315e73f admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:08:57.205277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1412/2815] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:08:57 2022] POST /identity/v3/users => generated 284 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:08:57.209446 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7687e8dd-ed54-4449-8140-f3805e1efb44 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:08:57.220072 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7687e8dd-ed54-4449-8140-f3805e1efb44 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:08:57.224970 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43ed72f-6bec-4bbc-bee9-e4e3351c9386 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:08:57.225563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43ed72f-6bec-4bbc-bee9-e4e3351c9386 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:57.225839 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43ed72f-6bec-4bbc-bee9-e4e3351c9386 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:57.226156 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43ed72f-6bec-4bbc-bee9-e4e3351c9386 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:57.226781 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43ed72f-6bec-4bbc-bee9-e4e3351c9386 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:08:57.228348 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43ed72f-6bec-4bbc-bee9-e4e3351c9386 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:08:57.238650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1404/2816] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:08:57 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:08:57.242704 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561b608c-6a75-4d21-adb9-5988c315e73f admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:08:57.252996 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561b608c-6a75-4d21-adb9-5988c315e73f admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:08:57.257673 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0368fa3-26a6-4abe-9712-9976e8cfaace admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:08:57.258275 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0368fa3-26a6-4abe-9712-9976e8cfaace admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:57.258593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0368fa3-26a6-4abe-9712-9976e8cfaace admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:57.259559 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0368fa3-26a6-4abe-9712-9976e8cfaace admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:57.260201 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0368fa3-26a6-4abe-9712-9976e8cfaace admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:08:57.261807 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0368fa3-26a6-4abe-9712-9976e8cfaace admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:08:57.272285 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1413/2817] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:08:57 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:08:57.276347 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43ed72f-6bec-4bbc-bee9-e4e3351c9386 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:08:57.286122 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43ed72f-6bec-4bbc-bee9-e4e3351c9386 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:08:57.291320 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2c0e5c-219a-4370-80fd-dd954ae85f52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:08:57.291967 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2c0e5c-219a-4370-80fd-dd954ae85f52 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:57.292285 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2c0e5c-219a-4370-80fd-dd954ae85f52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:57.292531 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2c0e5c-219a-4370-80fd-dd954ae85f52 admin admin] PATH_INFO: `/v3/projects/689a7be0863043e1ac79035e51cca2f1/users/072d0d01069b486399a714509e3e49a0/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:57.307983 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2c0e5c-219a-4370-80fd-dd954ae85f52 admin admin] RBAC: Authorizing `identity:create_grant(project_id=689a7be0863043e1ac79035e51cca2f1, user_id=072d0d01069b486399a714509e3e49a0, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:08:57.310805 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2c0e5c-219a-4370-80fd-dd954ae85f52 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:08:57.323846 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1405/2818] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:08:57 2022] PUT /identity/v3/projects/689a7be0863043e1ac79035e51cca2f1/users/072d0d01069b486399a714509e3e49a0/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:08:57.335023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19326a3f-7fd2-4279-b9a3-2355b472f42f None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:57.335305 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19326a3f-7fd2-4279-b9a3-2355b472f42f None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:57.335579 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19326a3f-7fd2-4279-b9a3-2355b472f42f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:57.364689 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-19326a3f-7fd2-4279-b9a3-2355b472f42f None None] MFA Rules not processed for user `072d0d01069b486399a714509e3e49a0`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:08:57.365927 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-19326a3f-7fd2-4279-b9a3-2355b472f42f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:08:57.428572 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1414/2819] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:08:57 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:08:57.439843 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2c0e5c-219a-4370-80fd-dd954ae85f52 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:08:57.441558 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-9b2c0e5c-219a-4370-80fd-dd954ae85f52 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:08:57.462021 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:08:57.503058 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2c0e5c-219a-4370-80fd-dd954ae85f52 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:08:57.506647 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63832a3d-406f-43d2-b8ae-973f956ece73 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:08:57.507275 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63832a3d-406f-43d2-b8ae-973f956ece73 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:08:57.507572 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63832a3d-406f-43d2-b8ae-973f956ece73 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:08:57.507839 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63832a3d-406f-43d2-b8ae-973f956ece73 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:08:57.516662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63832a3d-406f-43d2-b8ae-973f956ece73 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:08:57.518134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63832a3d-406f-43d2-b8ae-973f956ece73 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:08:57.527522 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1406/2820] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:08:57 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:09:05.884068 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19326a3f-7fd2-4279-b9a3-2355b472f42f None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:05.946488 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19326a3f-7fd2-4279-b9a3-2355b472f42f None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:05.950984 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c572e3-333e-4489-a09e-9de0d42ac139 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:05.951610 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c572e3-333e-4489-a09e-9de0d42ac139 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:05.951885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c572e3-333e-4489-a09e-9de0d42ac139 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:05.952156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c572e3-333e-4489-a09e-9de0d42ac139 admin admin] PATH_INFO: `/v3/users/072d0d01069b486399a714509e3e49a0` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:05.953257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c572e3-333e-4489-a09e-9de0d42ac139 admin admin] RBAC: Authorizing `identity:delete_user(user_id=072d0d01069b486399a714509e3e49a0)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:05.955195 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c572e3-333e-4489-a09e-9de0d42ac139 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:05.971582 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:09:06.001841 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d4c572e3-333e-4489-a09e-9de0d42ac139 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '072d0d01069b486399a714509e3e49a0'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:06.008510 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d4c572e3-333e-4489-a09e-9de0d42ac139 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '072d0d01069b486399a714509e3e49a0'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:06.020480 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d4c572e3-333e-4489-a09e-9de0d42ac139 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '072d0d01069b486399a714509e3e49a0'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:06.021028 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d4c572e3-333e-4489-a09e-9de0d42ac139 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '072d0d01069b486399a714509e3e49a0'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:06.028225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d4c572e3-333e-4489-a09e-9de0d42ac139 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '072d0d01069b486399a714509e3e49a0'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:06.034390 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1415/2821] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:09:05 2022] DELETE /identity/v3/users/072d0d01069b486399a714509e3e49a0 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:09:06.038953 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e513cc-ec01-4a97-97ec-d3ae99a90dca None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:06.039422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e513cc-ec01-4a97-97ec-d3ae99a90dca None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:06.039537 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e513cc-ec01-4a97-97ec-d3ae99a90dca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:06.058533 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-c5e513cc-ec01-4a97-97ec-d3ae99a90dca None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:09:06.059747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c5e513cc-ec01-4a97-97ec-d3ae99a90dca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:06.121286 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1407/2822] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:09:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:09:06.130980 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c572e3-333e-4489-a09e-9de0d42ac139 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:06.132431 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-d4c572e3-333e-4489-a09e-9de0d42ac139 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:06.192750 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c572e3-333e-4489-a09e-9de0d42ac139 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:06.196394 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d23cc79-eb3f-4e3b-a1b9-a84d91a49e3c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:06.196999 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d23cc79-eb3f-4e3b-a1b9-a84d91a49e3c service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:06.197278 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d23cc79-eb3f-4e3b-a1b9-a84d91a49e3c service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:06.197559 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d23cc79-eb3f-4e3b-a1b9-a84d91a49e3c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:06.206575 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d23cc79-eb3f-4e3b-a1b9-a84d91a49e3c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:06.207998 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d23cc79-eb3f-4e3b-a1b9-a84d91a49e3c service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:06.217968 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1416/2823] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:09:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:09:06.720225 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e513cc-ec01-4a97-97ec-d3ae99a90dca None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:06.721662 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c5e513cc-ec01-4a97-97ec-d3ae99a90dca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:06.733568 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e513cc-ec01-4a97-97ec-d3ae99a90dca None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:06.738204 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db32157-c51f-488a-bafb-8137e9ceac21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:06.738886 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db32157-c51f-488a-bafb-8137e9ceac21 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:06.739139 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db32157-c51f-488a-bafb-8137e9ceac21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:06.739405 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db32157-c51f-488a-bafb-8137e9ceac21 admin admin] PATH_INFO: `/v3/projects/689a7be0863043e1ac79035e51cca2f1` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:06.740693 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db32157-c51f-488a-bafb-8137e9ceac21 admin admin] RBAC: Authorizing `identity:delete_project(project_id=689a7be0863043e1ac79035e51cca2f1)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:06.742543 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db32157-c51f-488a-bafb-8137e9ceac21 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:06.789144 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-1db32157-c51f-488a-bafb-8137e9ceac21 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '689a7be0863043e1ac79035e51cca2f1', 'request_id': 'req-1db32157-c51f-488a-bafb-8137e9ceac21'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:06.794604 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-1db32157-c51f-488a-bafb-8137e9ceac21 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '689a7be0863043e1ac79035e51cca2f1', 'request_id': 'req-1db32157-c51f-488a-bafb-8137e9ceac21'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:06.802076 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-1db32157-c51f-488a-bafb-8137e9ceac21 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '689a7be0863043e1ac79035e51cca2f1', 'request_id': 'req-1db32157-c51f-488a-bafb-8137e9ceac21'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:06.813404 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-1db32157-c51f-488a-bafb-8137e9ceac21 admin admin] The token cache is being invalidate because project 689a7be0863043e1ac79035e51cca2f1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:09:06.813688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-1db32157-c51f-488a-bafb-8137e9ceac21 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:06.814358 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-1db32157-c51f-488a-bafb-8137e9ceac21 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:06.815837 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1408/2824] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:09:06 2022] DELETE /identity/v3/projects/689a7be0863043e1ac79035e51cca2f1 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:09:06.820798 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bebeaf-b8d9-448a-8533-d44abed9405d None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:06.821107 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bebeaf-b8d9-448a-8533-d44abed9405d None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:06.821383 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bebeaf-b8d9-448a-8533-d44abed9405d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:06.842486 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-d8bebeaf-b8d9-448a-8533-d44abed9405d None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:09:06.843939 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-d8bebeaf-b8d9-448a-8533-d44abed9405d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:06.901312 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1417/2825] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:09:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:09:06.909189 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843a59ea-51d1-4888-ac5a-b55c1eac8e05 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:06.909475 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843a59ea-51d1-4888-ac5a-b55c1eac8e05 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:06.909747 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843a59ea-51d1-4888-ac5a-b55c1eac8e05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:06.928961 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-843a59ea-51d1-4888-ac5a-b55c1eac8e05 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:09:06.930151 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-843a59ea-51d1-4888-ac5a-b55c1eac8e05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:06.938408 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1409/2826] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:09:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:09:06.942512 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8bebeaf-b8d9-448a-8533-d44abed9405d None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:06.952507 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8bebeaf-b8d9-448a-8533-d44abed9405d None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:06.956988 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e7a92d-14b5-41b8-9aef-86b0ad799f91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:06.957531 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e7a92d-14b5-41b8-9aef-86b0ad799f91 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:06.957805 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e7a92d-14b5-41b8-9aef-86b0ad799f91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:06.958098 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e7a92d-14b5-41b8-9aef-86b0ad799f91 admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:06.958753 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e7a92d-14b5-41b8-9aef-86b0ad799f91 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:09:06.959070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e7a92d-14b5-41b8-9aef-86b0ad799f91 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:06.960629 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e7a92d-14b5-41b8-9aef-86b0ad799f91 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:06.973544 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1418/2827] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:09:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 01:09:06.977820 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843a59ea-51d1-4888-ac5a-b55c1eac8e05 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:06.987640 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843a59ea-51d1-4888-ac5a-b55c1eac8e05 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:06.992324 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0984fb70-d1cc-4362-a676-6f048956e7d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:06.992899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0984fb70-d1cc-4362-a676-6f048956e7d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:06.993174 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0984fb70-d1cc-4362-a676-6f048956e7d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:06.993442 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0984fb70-d1cc-4362-a676-6f048956e7d5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:06.994090 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0984fb70-d1cc-4362-a676-6f048956e7d5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:06.996579 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0984fb70-d1cc-4362-a676-6f048956e7d5 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:07.023961 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1410/2828] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:09:06 2022] POST /identity/v3/projects => generated 365 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:09:07.027879 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e7a92d-14b5-41b8-9aef-86b0ad799f91 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:07.037738 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e7a92d-14b5-41b8-9aef-86b0ad799f91 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:07.057997 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:09:07.100187 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fc6d70-0178-4eb6-9966-7e4cc5198f92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:07.100927 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fc6d70-0178-4eb6-9966-7e4cc5198f92 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:07.101257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fc6d70-0178-4eb6-9966-7e4cc5198f92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:07.101532 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fc6d70-0178-4eb6-9966-7e4cc5198f92 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:07.102264 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fc6d70-0178-4eb6-9966-7e4cc5198f92 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:07.104819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fc6d70-0178-4eb6-9966-7e4cc5198f92 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:07.127621 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1419/2829] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:09:07 2022] POST /identity/v3/users => generated 283 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:09:07.131877 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0984fb70-d1cc-4362-a676-6f048956e7d5 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:07.144111 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0984fb70-d1cc-4362-a676-6f048956e7d5 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:07.148824 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d79292-a1ba-4af9-bf02-aa8c997f7b6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:07.149414 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d79292-a1ba-4af9-bf02-aa8c997f7b6b admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:07.149685 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d79292-a1ba-4af9-bf02-aa8c997f7b6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:07.149954 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d79292-a1ba-4af9-bf02-aa8c997f7b6b admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:07.150623 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d79292-a1ba-4af9-bf02-aa8c997f7b6b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:07.152251 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d79292-a1ba-4af9-bf02-aa8c997f7b6b admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:07.162231 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1411/2830] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:09:07 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:09:07.166225 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fc6d70-0178-4eb6-9966-7e4cc5198f92 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:07.176602 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fc6d70-0178-4eb6-9966-7e4cc5198f92 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:07.181161 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe1a005-d5b8-4669-be20-08bda3733322 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:07.181748 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe1a005-d5b8-4669-be20-08bda3733322 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:07.182023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe1a005-d5b8-4669-be20-08bda3733322 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:07.182889 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe1a005-d5b8-4669-be20-08bda3733322 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:07.183545 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe1a005-d5b8-4669-be20-08bda3733322 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:07.185134 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe1a005-d5b8-4669-be20-08bda3733322 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:07.195765 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1420/2831] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:09:07 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:09:07.199968 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d79292-a1ba-4af9-bf02-aa8c997f7b6b admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:07.206593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:09:07.210957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d79292-a1ba-4af9-bf02-aa8c997f7b6b admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:07.217532 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aeb75c8-ef15-4620-8775-8c085d418c42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:07.218474 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aeb75c8-ef15-4620-8775-8c085d418c42 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:07.218921 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aeb75c8-ef15-4620-8775-8c085d418c42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:07.219253 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aeb75c8-ef15-4620-8775-8c085d418c42 admin admin] PATH_INFO: `/v3/projects/4652ddbd662648fb8769ee854cc94bf8/users/f67b88bdcb964ad3bda0bd8ec972fa49/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:07.234612 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aeb75c8-ef15-4620-8775-8c085d418c42 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4652ddbd662648fb8769ee854cc94bf8, user_id=f67b88bdcb964ad3bda0bd8ec972fa49, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:07.237182 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aeb75c8-ef15-4620-8775-8c085d418c42 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:07.250528 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1412/2832] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:09:07 2022] PUT /identity/v3/projects/4652ddbd662648fb8769ee854cc94bf8/users/f67b88bdcb964ad3bda0bd8ec972fa49/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:09:07.262195 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b088294-2bee-442f-8a05-845a4732ce61 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:07.262498 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b088294-2bee-442f-8a05-845a4732ce61 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:07.262774 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b088294-2bee-442f-8a05-845a4732ce61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:07.289545 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-3b088294-2bee-442f-8a05-845a4732ce61 None None] MFA Rules not processed for user `f67b88bdcb964ad3bda0bd8ec972fa49`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:09:07.290759 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3b088294-2bee-442f-8a05-845a4732ce61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:07.352985 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1421/2833] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:09:07 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:09:07.358266 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aeb75c8-ef15-4620-8775-8c085d418c42 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:07.422380 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aeb75c8-ef15-4620-8775-8c085d418c42 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:07.427038 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6643272-2c43-4b44-a926-ad0e518f4394 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:07.427636 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6643272-2c43-4b44-a926-ad0e518f4394 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:07.427911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6643272-2c43-4b44-a926-ad0e518f4394 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:07.428181 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6643272-2c43-4b44-a926-ad0e518f4394 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:07.428801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6643272-2c43-4b44-a926-ad0e518f4394 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:07.430704 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6643272-2c43-4b44-a926-ad0e518f4394 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:07.456497 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1413/2834] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:09:07 2022] POST /identity/v3/projects => generated 365 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:09:07.460630 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b088294-2bee-442f-8a05-845a4732ce61 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:07.472250 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b088294-2bee-442f-8a05-845a4732ce61 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:07.527634 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd236eb7-583d-46bf-95c6-ff939f61e5b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:07.528251 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd236eb7-583d-46bf-95c6-ff939f61e5b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:07.528526 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd236eb7-583d-46bf-95c6-ff939f61e5b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:07.528800 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd236eb7-583d-46bf-95c6-ff939f61e5b5 admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:07.529446 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd236eb7-583d-46bf-95c6-ff939f61e5b5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:07.531390 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd236eb7-583d-46bf-95c6-ff939f61e5b5 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:07.555949 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1422/2835] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:09:07 2022] POST /identity/v3/users => generated 289 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:09:07.560438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6643272-2c43-4b44-a926-ad0e518f4394 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:07.570934 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6643272-2c43-4b44-a926-ad0e518f4394 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:07.575544 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b7465d-4eae-4975-b892-99443cd5d68a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:07.576135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b7465d-4eae-4975-b892-99443cd5d68a admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:07.576411 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b7465d-4eae-4975-b892-99443cd5d68a admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:07.576679 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b7465d-4eae-4975-b892-99443cd5d68a admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:07.577309 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b7465d-4eae-4975-b892-99443cd5d68a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:07.578925 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b7465d-4eae-4975-b892-99443cd5d68a admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:07.589960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1414/2836] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:09:07 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:09:07.594555 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd236eb7-583d-46bf-95c6-ff939f61e5b5 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:07.605218 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd236eb7-583d-46bf-95c6-ff939f61e5b5 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:07.609842 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40a96aa-6d24-467d-b484-5fe3169fc42f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:07.610547 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40a96aa-6d24-467d-b484-5fe3169fc42f admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:07.610830 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40a96aa-6d24-467d-b484-5fe3169fc42f admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:07.611130 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40a96aa-6d24-467d-b484-5fe3169fc42f admin admin] PATH_INFO: `/v3/projects/4bf69e2139fc4408a8af9a0e2c7fa931/users/e92ea136796947348f322e55bef98685/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:07.625674 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40a96aa-6d24-467d-b484-5fe3169fc42f admin admin] RBAC: Authorizing `identity:create_grant(project_id=4bf69e2139fc4408a8af9a0e2c7fa931, user_id=e92ea136796947348f322e55bef98685, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:07.628374 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40a96aa-6d24-467d-b484-5fe3169fc42f admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:07.636556 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:09:07.642739 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1423/2837] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:09:07 2022] PUT /identity/v3/projects/4bf69e2139fc4408a8af9a0e2c7fa931/users/e92ea136796947348f322e55bef98685/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:09:07.647069 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b7465d-4eae-4975-b892-99443cd5d68a admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:07.657910 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b7465d-4eae-4975-b892-99443cd5d68a admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:07.712049 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6115297e-3fd1-4519-a9a6-3fe5dc36bc77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:07.712643 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6115297e-3fd1-4519-a9a6-3fe5dc36bc77 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:07.712919 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6115297e-3fd1-4519-a9a6-3fe5dc36bc77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:07.713189 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6115297e-3fd1-4519-a9a6-3fe5dc36bc77 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:07.714324 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6115297e-3fd1-4519-a9a6-3fe5dc36bc77 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:07.715952 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6115297e-3fd1-4519-a9a6-3fe5dc36bc77 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:07.725936 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1415/2838] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:09:07 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:09:07.730859 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40a96aa-6d24-467d-b484-5fe3169fc42f admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:07.741045 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40a96aa-6d24-467d-b484-5fe3169fc42f admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:07.745539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fbc4df-9c1d-49d3-9730-b4755bc9254a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:07.746176 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fbc4df-9c1d-49d3-9730-b4755bc9254a admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:07.746474 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fbc4df-9c1d-49d3-9730-b4755bc9254a admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:07.746831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fbc4df-9c1d-49d3-9730-b4755bc9254a admin admin] PATH_INFO: `/v3/projects/4bf69e2139fc4408a8af9a0e2c7fa931/users/e92ea136796947348f322e55bef98685/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:07.748660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fbc4df-9c1d-49d3-9730-b4755bc9254a admin admin] RBAC: Authorizing `identity:create_grant(project_id=4bf69e2139fc4408a8af9a0e2c7fa931, user_id=e92ea136796947348f322e55bef98685, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:07.751360 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fbc4df-9c1d-49d3-9730-b4755bc9254a admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:07.761803 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1424/2839] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:09:07 2022] PUT /identity/v3/projects/4bf69e2139fc4408a8af9a0e2c7fa931/users/e92ea136796947348f322e55bef98685/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:09:07.772017 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36a4749-2583-4268-a656-09ebf8e882e7 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:07.772302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36a4749-2583-4268-a656-09ebf8e882e7 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:07.772573 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36a4749-2583-4268-a656-09ebf8e882e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:07.800570 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-e36a4749-2583-4268-a656-09ebf8e882e7 None None] MFA Rules not processed for user `e92ea136796947348f322e55bef98685`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:09:07.801780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-e36a4749-2583-4268-a656-09ebf8e882e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:07.842202 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:09:07.877278 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1416/2840] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:09:07 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:09:07.889325 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fbc4df-9c1d-49d3-9730-b4755bc9254a admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:07.891100 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-50fbc4df-9c1d-49d3-9730-b4755bc9254a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:07.949553 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fbc4df-9c1d-49d3-9730-b4755bc9254a admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:07.953283 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30132db5-4177-4ea7-bc25-2e1bac36dbb9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:07.953887 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30132db5-4177-4ea7-bc25-2e1bac36dbb9 service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:07.954163 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30132db5-4177-4ea7-bc25-2e1bac36dbb9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:07.954439 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30132db5-4177-4ea7-bc25-2e1bac36dbb9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:07.963429 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30132db5-4177-4ea7-bc25-2e1bac36dbb9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:07.965578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30132db5-4177-4ea7-bc25-2e1bac36dbb9 service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:07.975269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1425/2841] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:09:07 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:09:08.393630 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36a4749-2583-4268-a656-09ebf8e882e7 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:08.455723 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36a4749-2583-4268-a656-09ebf8e882e7 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:08.460252 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ed8338-1b7e-4bef-8f6e-e4d3ce117afe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:08.460876 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ed8338-1b7e-4bef-8f6e-e4d3ce117afe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:08.461150 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ed8338-1b7e-4bef-8f6e-e4d3ce117afe admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:08.461419 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ed8338-1b7e-4bef-8f6e-e4d3ce117afe admin admin] PATH_INFO: `/v3/users/f67b88bdcb964ad3bda0bd8ec972fa49` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:08.462593 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0ed8338-1b7e-4bef-8f6e-e4d3ce117afe admin admin] RBAC: Authorizing `identity:delete_user(user_id=f67b88bdcb964ad3bda0bd8ec972fa49)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:08.464488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0ed8338-1b7e-4bef-8f6e-e4d3ce117afe admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:08.509906 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c0ed8338-1b7e-4bef-8f6e-e4d3ce117afe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f67b88bdcb964ad3bda0bd8ec972fa49'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:08.518984 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c0ed8338-1b7e-4bef-8f6e-e4d3ce117afe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f67b88bdcb964ad3bda0bd8ec972fa49'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:08.521438 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c0ed8338-1b7e-4bef-8f6e-e4d3ce117afe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f67b88bdcb964ad3bda0bd8ec972fa49'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:08.522130 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c0ed8338-1b7e-4bef-8f6e-e4d3ce117afe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f67b88bdcb964ad3bda0bd8ec972fa49'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:08.530129 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-c0ed8338-1b7e-4bef-8f6e-e4d3ce117afe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f67b88bdcb964ad3bda0bd8ec972fa49'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:08.547999 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1417/2842] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:09:08 2022] DELETE /identity/v3/users/f67b88bdcb964ad3bda0bd8ec972fa49 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:09:08.552767 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830485c0-7a10-4d06-acc3-440659d735a7 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:08.553102 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830485c0-7a10-4d06-acc3-440659d735a7 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:08.553379 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830485c0-7a10-4d06-acc3-440659d735a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:08.572930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-830485c0-7a10-4d06-acc3-440659d735a7 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:09:08.574037 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-830485c0-7a10-4d06-acc3-440659d735a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:08.640305 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1426/2843] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:09:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:09:08.650654 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ed8338-1b7e-4bef-8f6e-e4d3ce117afe admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:08.652072 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-c0ed8338-1b7e-4bef-8f6e-e4d3ce117afe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:08.708604 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ed8338-1b7e-4bef-8f6e-e4d3ce117afe admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:08.712230 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e069b9a7-3c9b-4b08-a41f-7c97b55b1534 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:08.712825 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e069b9a7-3c9b-4b08-a41f-7c97b55b1534 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:08.713101 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e069b9a7-3c9b-4b08-a41f-7c97b55b1534 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:08.713370 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e069b9a7-3c9b-4b08-a41f-7c97b55b1534 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:08.722521 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e069b9a7-3c9b-4b08-a41f-7c97b55b1534 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:08.723928 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e069b9a7-3c9b-4b08-a41f-7c97b55b1534 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:08.727728 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:09:08.736239 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1418/2844] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:09:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:09:08.993946 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830485c0-7a10-4d06-acc3-440659d735a7 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:08.995488 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-830485c0-7a10-4d06-acc3-440659d735a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:09.009341 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830485c0-7a10-4d06-acc3-440659d735a7 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:09.014091 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf092b9-6e86-4956-b9f2-78af7231f130 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:09.014793 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf092b9-6e86-4956-b9f2-78af7231f130 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:09.015099 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf092b9-6e86-4956-b9f2-78af7231f130 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:09.015386 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf092b9-6e86-4956-b9f2-78af7231f130 admin admin] PATH_INFO: `/v3/projects/4652ddbd662648fb8769ee854cc94bf8` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:09.016787 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf092b9-6e86-4956-b9f2-78af7231f130 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4652ddbd662648fb8769ee854cc94bf8)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:09.019806 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf092b9-6e86-4956-b9f2-78af7231f130 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:09.067853 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7cf092b9-6e86-4956-b9f2-78af7231f130 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4652ddbd662648fb8769ee854cc94bf8', 'request_id': 'req-7cf092b9-6e86-4956-b9f2-78af7231f130'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:09.073539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7cf092b9-6e86-4956-b9f2-78af7231f130 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4652ddbd662648fb8769ee854cc94bf8', 'request_id': 'req-7cf092b9-6e86-4956-b9f2-78af7231f130'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:09.081172 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7cf092b9-6e86-4956-b9f2-78af7231f130 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4652ddbd662648fb8769ee854cc94bf8', 'request_id': 'req-7cf092b9-6e86-4956-b9f2-78af7231f130'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:09.093011 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7cf092b9-6e86-4956-b9f2-78af7231f130 admin admin] The token cache is being invalidate because project 4652ddbd662648fb8769ee854cc94bf8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:09:09.093326 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7cf092b9-6e86-4956-b9f2-78af7231f130 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:09.094038 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-7cf092b9-6e86-4956-b9f2-78af7231f130 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:09.096025 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1427/2845] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:09:08 2022] DELETE /identity/v3/projects/4652ddbd662648fb8769ee854cc94bf8 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:09:09.100298 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e069b9a7-3c9b-4b08-a41f-7c97b55b1534 service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:09.110794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e069b9a7-3c9b-4b08-a41f-7c97b55b1534 service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:09.166430 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e588a077-59f1-4e9c-9342-5e01576cb794 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:09.167086 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e588a077-59f1-4e9c-9342-5e01576cb794 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:09.167367 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e588a077-59f1-4e9c-9342-5e01576cb794 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:09.167664 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e588a077-59f1-4e9c-9342-5e01576cb794 admin admin] PATH_INFO: `/v3/users/e92ea136796947348f322e55bef98685` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:09.169422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e588a077-59f1-4e9c-9342-5e01576cb794 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e92ea136796947348f322e55bef98685)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:09.171428 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e588a077-59f1-4e9c-9342-5e01576cb794 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:09.217994 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-e588a077-59f1-4e9c-9342-5e01576cb794 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e92ea136796947348f322e55bef98685'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:09.226827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-e588a077-59f1-4e9c-9342-5e01576cb794 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e92ea136796947348f322e55bef98685'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:09.227443 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-e588a077-59f1-4e9c-9342-5e01576cb794 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e92ea136796947348f322e55bef98685'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:09.228051 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-e588a077-59f1-4e9c-9342-5e01576cb794 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e92ea136796947348f322e55bef98685'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:09.236731 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-e588a077-59f1-4e9c-9342-5e01576cb794 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e92ea136796947348f322e55bef98685'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:09.254421 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1419/2846] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:09:09 2022] DELETE /identity/v3/users/e92ea136796947348f322e55bef98685 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:09:09.526685 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf092b9-6e86-4956-b9f2-78af7231f130 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:09.529836 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-7cf092b9-6e86-4956-b9f2-78af7231f130 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:09.545828 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf092b9-6e86-4956-b9f2-78af7231f130 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:09.554960 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:09:09.607940 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af8118f-7c64-451e-bdfa-09af38a2a988 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:09.608506 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af8118f-7c64-451e-bdfa-09af38a2a988 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:09.608784 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af8118f-7c64-451e-bdfa-09af38a2a988 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:09.609053 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af8118f-7c64-451e-bdfa-09af38a2a988 admin admin] PATH_INFO: `/v3/projects/4bf69e2139fc4408a8af9a0e2c7fa931` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:09.610200 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af8118f-7c64-451e-bdfa-09af38a2a988 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4bf69e2139fc4408a8af9a0e2c7fa931)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:09.612188 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af8118f-7c64-451e-bdfa-09af38a2a988 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:09.658197 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-3af8118f-7c64-451e-bdfa-09af38a2a988 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4bf69e2139fc4408a8af9a0e2c7fa931', 'request_id': 'req-3af8118f-7c64-451e-bdfa-09af38a2a988'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:09.663757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-3af8118f-7c64-451e-bdfa-09af38a2a988 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4bf69e2139fc4408a8af9a0e2c7fa931', 'request_id': 'req-3af8118f-7c64-451e-bdfa-09af38a2a988'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:09.670999 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-3af8118f-7c64-451e-bdfa-09af38a2a988 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4bf69e2139fc4408a8af9a0e2c7fa931', 'request_id': 'req-3af8118f-7c64-451e-bdfa-09af38a2a988'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:09.687288 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-3af8118f-7c64-451e-bdfa-09af38a2a988 admin admin] The token cache is being invalidate because project 4bf69e2139fc4408a8af9a0e2c7fa931 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:09:09.687757 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-3af8118f-7c64-451e-bdfa-09af38a2a988 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:09.688832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-3af8118f-7c64-451e-bdfa-09af38a2a988 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:09.690156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1428/2847] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:09:09 2022] DELETE /identity/v3/projects/4bf69e2139fc4408a8af9a0e2c7fa931 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 01:09:09.699394 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c024f0b-aea5-415a-bb7a-e8956d27bd7c None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:09.699924 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c024f0b-aea5-415a-bb7a-e8956d27bd7c None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:09.700052 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c024f0b-aea5-415a-bb7a-e8956d27bd7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:09.722065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-0c024f0b-aea5-415a-bb7a-e8956d27bd7c None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:09:09.723306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-0c024f0b-aea5-415a-bb7a-e8956d27bd7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:09.782758 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1420/2848] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:09:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:09:09.790816 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d77543-9479-4819-8b75-0ff0ec1f9009 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:09.791108 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d77543-9479-4819-8b75-0ff0ec1f9009 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:09.791385 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d77543-9479-4819-8b75-0ff0ec1f9009 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:09.814634 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-24d77543-9479-4819-8b75-0ff0ec1f9009 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:09:09.816536 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-24d77543-9479-4819-8b75-0ff0ec1f9009 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:09.826490 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1429/2849] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:09:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:09:09.830784 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c024f0b-aea5-415a-bb7a-e8956d27bd7c None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:09.841487 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c024f0b-aea5-415a-bb7a-e8956d27bd7c None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:09.848096 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad293b6b-9e72-4bc5-874d-c3e17039adf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:09.848695 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad293b6b-9e72-4bc5-874d-c3e17039adf5 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:09.849135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad293b6b-9e72-4bc5-874d-c3e17039adf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:09.849416 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad293b6b-9e72-4bc5-874d-c3e17039adf5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:09.850067 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad293b6b-9e72-4bc5-874d-c3e17039adf5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:09:09.850371 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad293b6b-9e72-4bc5-874d-c3e17039adf5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:09.852006 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad293b6b-9e72-4bc5-874d-c3e17039adf5 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:09.864255 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1421/2850] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:09:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 01:09:09.868727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d77543-9479-4819-8b75-0ff0ec1f9009 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:09.878477 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d77543-9479-4819-8b75-0ff0ec1f9009 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:09.884146 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb61641-6049-4160-a144-687d7fd575a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:09.884767 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb61641-6049-4160-a144-687d7fd575a1 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:09.885042 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb61641-6049-4160-a144-687d7fd575a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:09.885310 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb61641-6049-4160-a144-687d7fd575a1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:09.885953 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb61641-6049-4160-a144-687d7fd575a1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:09.888515 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb61641-6049-4160-a144-687d7fd575a1 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:09.917730 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1430/2851] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:09:09 2022] POST /identity/v3/projects => generated 383 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:09:09.922480 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad293b6b-9e72-4bc5-874d-c3e17039adf5 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:09.929277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:09:09.933469 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad293b6b-9e72-4bc5-874d-c3e17039adf5 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:09.993229 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dd717f-2acc-481d-8aa9-5523e93d0488 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:09.993843 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dd717f-2acc-481d-8aa9-5523e93d0488 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:09.994112 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dd717f-2acc-481d-8aa9-5523e93d0488 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:09.994380 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dd717f-2acc-481d-8aa9-5523e93d0488 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:09.995055 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78dd717f-2acc-481d-8aa9-5523e93d0488 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:09.997582 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78dd717f-2acc-481d-8aa9-5523e93d0488 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:10.018750 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1422/2852] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:09:09 2022] POST /identity/v3/users => generated 292 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:09:10.023659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb61641-6049-4160-a144-687d7fd575a1 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:10.034248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb61641-6049-4160-a144-687d7fd575a1 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:10.038993 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5796d89-babb-4b10-a624-90af0650db56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:10.039593 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5796d89-babb-4b10-a624-90af0650db56 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:10.039868 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5796d89-babb-4b10-a624-90af0650db56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:10.040137 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5796d89-babb-4b10-a624-90af0650db56 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:10.040775 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5796d89-babb-4b10-a624-90af0650db56 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:10.042536 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5796d89-babb-4b10-a624-90af0650db56 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:10.053500 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1431/2853] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:09:10 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:09:10.057617 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dd717f-2acc-481d-8aa9-5523e93d0488 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:10.068553 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dd717f-2acc-481d-8aa9-5523e93d0488 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:10.073205 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa0f8b5-a42e-4ae8-a8c7-a6cb109bfca9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:10.073799 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa0f8b5-a42e-4ae8-a8c7-a6cb109bfca9 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:10.074075 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa0f8b5-a42e-4ae8-a8c7-a6cb109bfca9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:10.074341 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa0f8b5-a42e-4ae8-a8c7-a6cb109bfca9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:10.075011 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa0f8b5-a42e-4ae8-a8c7-a6cb109bfca9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:10.076721 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa0f8b5-a42e-4ae8-a8c7-a6cb109bfca9 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:10.087585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1423/2854] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:09:10 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:09:10.092456 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5796d89-babb-4b10-a624-90af0650db56 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:10.103750 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5796d89-babb-4b10-a624-90af0650db56 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:10.108904 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490ac919-4edc-48ea-9e9f-e4ec7b9e35e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:10.109571 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490ac919-4edc-48ea-9e9f-e4ec7b9e35e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:10.109925 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490ac919-4edc-48ea-9e9f-e4ec7b9e35e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:10.110203 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490ac919-4edc-48ea-9e9f-e4ec7b9e35e4 admin admin] PATH_INFO: `/v3/projects/56ae2847ac8442a587478b17b54953f6/users/2bb7c030065d4255a9a8243af1a1e926/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:10.126257 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490ac919-4edc-48ea-9e9f-e4ec7b9e35e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=56ae2847ac8442a587478b17b54953f6, user_id=2bb7c030065d4255a9a8243af1a1e926, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:10.128938 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490ac919-4edc-48ea-9e9f-e4ec7b9e35e4 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:10.142761 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1432/2855] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:09:10 2022] PUT /identity/v3/projects/56ae2847ac8442a587478b17b54953f6/users/2bb7c030065d4255a9a8243af1a1e926/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 01:09:10.154301 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01edd810-a66e-4b12-ba4b-b21c976fb4f3 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:10.154642 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01edd810-a66e-4b12-ba4b-b21c976fb4f3 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:10.154924 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01edd810-a66e-4b12-ba4b-b21c976fb4f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:10.188031 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-01edd810-a66e-4b12-ba4b-b21c976fb4f3 None None] MFA Rules not processed for user `2bb7c030065d4255a9a8243af1a1e926`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:09:10.189296 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-01edd810-a66e-4b12-ba4b-b21c976fb4f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:10.254102 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1424/2856] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:09:10 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:09:10.258691 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490ac919-4edc-48ea-9e9f-e4ec7b9e35e4 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:10.304384 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:09:10.327242 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490ac919-4edc-48ea-9e9f-e4ec7b9e35e4 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:10.331926 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d06309-8993-48e3-9b15-5c9d479e7a65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:10.332538 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d06309-8993-48e3-9b15-5c9d479e7a65 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:10.332819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d06309-8993-48e3-9b15-5c9d479e7a65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:10.333095 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d06309-8993-48e3-9b15-5c9d479e7a65 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:10.333728 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d06309-8993-48e3-9b15-5c9d479e7a65 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:10.335677 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d06309-8993-48e3-9b15-5c9d479e7a65 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:10.365444 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1433/2857] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:09:10 2022] POST /identity/v3/projects => generated 385 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:09:10.369434 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01edd810-a66e-4b12-ba4b-b21c976fb4f3 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:10.380763 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01edd810-a66e-4b12-ba4b-b21c976fb4f3 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:10.437760 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad02e045-0ea4-46a0-a607-d1eb9523629c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:10.438385 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad02e045-0ea4-46a0-a607-d1eb9523629c admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:10.438701 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad02e045-0ea4-46a0-a607-d1eb9523629c admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:10.438972 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad02e045-0ea4-46a0-a607-d1eb9523629c admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:10.439609 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad02e045-0ea4-46a0-a607-d1eb9523629c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:10.441544 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad02e045-0ea4-46a0-a607-d1eb9523629c admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:10.463433 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1425/2858] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:09:10 2022] POST /identity/v3/users => generated 299 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:09:10.468316 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d06309-8993-48e3-9b15-5c9d479e7a65 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:10.478894 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d06309-8993-48e3-9b15-5c9d479e7a65 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:10.484273 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214c5940-46c9-4c2d-ba3e-ac6f6e5d1d79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:10.484981 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214c5940-46c9-4c2d-ba3e-ac6f6e5d1d79 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:10.485269 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214c5940-46c9-4c2d-ba3e-ac6f6e5d1d79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:10.485671 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214c5940-46c9-4c2d-ba3e-ac6f6e5d1d79 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:10.486833 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214c5940-46c9-4c2d-ba3e-ac6f6e5d1d79 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:10.488603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214c5940-46c9-4c2d-ba3e-ac6f6e5d1d79 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:10.499651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1434/2859] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:09:10 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:09:10.503727 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad02e045-0ea4-46a0-a607-d1eb9523629c admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:10.508960 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:09:10.517078 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad02e045-0ea4-46a0-a607-d1eb9523629c admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:10.523940 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc2a5a8-def8-492a-9dcd-055c5c4d0dc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:10.524994 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc2a5a8-def8-492a-9dcd-055c5c4d0dc9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:10.525523 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc2a5a8-def8-492a-9dcd-055c5c4d0dc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:10.525984 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc2a5a8-def8-492a-9dcd-055c5c4d0dc9 admin admin] PATH_INFO: `/v3/projects/50a05a71f71a490589e659b4290b03b5/users/b282704274d545ac9d203d6cbabb8207/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:10.545714 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc2a5a8-def8-492a-9dcd-055c5c4d0dc9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=50a05a71f71a490589e659b4290b03b5, user_id=b282704274d545ac9d203d6cbabb8207, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:10.548377 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc2a5a8-def8-492a-9dcd-055c5c4d0dc9 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:10.561911 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1426/2860] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:09:10 2022] PUT /identity/v3/projects/50a05a71f71a490589e659b4290b03b5/users/b282704274d545ac9d203d6cbabb8207/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:09:10.565929 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214c5940-46c9-4c2d-ba3e-ac6f6e5d1d79 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:10.575854 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214c5940-46c9-4c2d-ba3e-ac6f6e5d1d79 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:10.631417 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7d6ab4-8f47-4c64-bfbb-39a787f0ec47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:10.631947 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7d6ab4-8f47-4c64-bfbb-39a787f0ec47 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:10.632221 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7d6ab4-8f47-4c64-bfbb-39a787f0ec47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:10.632489 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7d6ab4-8f47-4c64-bfbb-39a787f0ec47 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:10.633236 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7d6ab4-8f47-4c64-bfbb-39a787f0ec47 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:10.634948 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7d6ab4-8f47-4c64-bfbb-39a787f0ec47 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:10.644788 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1435/2861] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:09:10 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:09:10.649035 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc2a5a8-def8-492a-9dcd-055c5c4d0dc9 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:10.659869 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc2a5a8-def8-492a-9dcd-055c5c4d0dc9 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:10.664396 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095a7213-cb43-48b1-9d42-662fdb8b5434 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:10.665032 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095a7213-cb43-48b1-9d42-662fdb8b5434 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:10.665306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095a7213-cb43-48b1-9d42-662fdb8b5434 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:10.665574 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095a7213-cb43-48b1-9d42-662fdb8b5434 admin admin] PATH_INFO: `/v3/projects/50a05a71f71a490589e659b4290b03b5/users/b282704274d545ac9d203d6cbabb8207/roles/1e04c9fea65244d28e1e59fc4c64f32f` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:10.667483 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095a7213-cb43-48b1-9d42-662fdb8b5434 admin admin] RBAC: Authorizing `identity:create_grant(project_id=50a05a71f71a490589e659b4290b03b5, user_id=b282704274d545ac9d203d6cbabb8207, role_id=1e04c9fea65244d28e1e59fc4c64f32f)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:10.670727 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095a7213-cb43-48b1-9d42-662fdb8b5434 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:10.680947 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1427/2862] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:09:10 2022] PUT /identity/v3/projects/50a05a71f71a490589e659b4290b03b5/users/b282704274d545ac9d203d6cbabb8207/roles/1e04c9fea65244d28e1e59fc4c64f32f => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:09:10.691038 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a038fae-0024-4627-922b-1611f6a009cd None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:10.691322 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a038fae-0024-4627-922b-1611f6a009cd None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:10.691594 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a038fae-0024-4627-922b-1611f6a009cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:10.721428 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-6a038fae-0024-4627-922b-1611f6a009cd None None] MFA Rules not processed for user `b282704274d545ac9d203d6cbabb8207`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:09:10.722669 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-6a038fae-0024-4627-922b-1611f6a009cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:10.789512 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1436/2863] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:09:10 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:09:10.800408 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095a7213-cb43-48b1-9d42-662fdb8b5434 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:10.802147 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-095a7213-cb43-48b1-9d42-662fdb8b5434 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:10.858712 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095a7213-cb43-48b1-9d42-662fdb8b5434 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:10.862263 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ce934a-d47b-4e09-a40f-a7cd524d642a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:10.862883 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ce934a-d47b-4e09-a40f-a7cd524d642a service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:10.863159 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ce934a-d47b-4e09-a40f-a7cd524d642a service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:10.863424 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ce934a-d47b-4e09-a40f-a7cd524d642a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:10.919363 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ce934a-d47b-4e09-a40f-a7cd524d642a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:10.920702 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ce934a-d47b-4e09-a40f-a7cd524d642a service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:10.929416 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1428/2864] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:09:10 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:09:15.177236 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a038fae-0024-4627-922b-1611f6a009cd None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:15.178663 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-6a038fae-0024-4627-922b-1611f6a009cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:15.190539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a038fae-0024-4627-922b-1611f6a009cd None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:15.194170 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f1c2fc-acde-4441-b76e-18f82c9d44e9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:15.194806 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f1c2fc-acde-4441-b76e-18f82c9d44e9 service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:15.195099 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f1c2fc-acde-4441-b76e-18f82c9d44e9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:15.195371 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f1c2fc-acde-4441-b76e-18f82c9d44e9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:15.204659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f1c2fc-acde-4441-b76e-18f82c9d44e9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:15.206080 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f1c2fc-acde-4441-b76e-18f82c9d44e9 service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:15.215504 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1437/2865] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:09:15 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:09:20.088851 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ce934a-d47b-4e09-a40f-a7cd524d642a service cinder] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:20.151462 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ce934a-d47b-4e09-a40f-a7cd524d642a service cinder] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:20.156026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c11d49-0c72-4a6f-9fdb-8c9a2adea470 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:20.156650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c11d49-0c72-4a6f-9fdb-8c9a2adea470 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:20.156931 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c11d49-0c72-4a6f-9fdb-8c9a2adea470 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:20.157196 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c11d49-0c72-4a6f-9fdb-8c9a2adea470 admin admin] PATH_INFO: `/v3/users/2bb7c030065d4255a9a8243af1a1e926` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:20.158309 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c11d49-0c72-4a6f-9fdb-8c9a2adea470 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2bb7c030065d4255a9a8243af1a1e926)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:20.160257 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c11d49-0c72-4a6f-9fdb-8c9a2adea470 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:20.180981 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:09:20.206795 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-f8c11d49-0c72-4a6f-9fdb-8c9a2adea470 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2bb7c030065d4255a9a8243af1a1e926'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:20.213818 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-f8c11d49-0c72-4a6f-9fdb-8c9a2adea470 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2bb7c030065d4255a9a8243af1a1e926'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:20.214434 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-f8c11d49-0c72-4a6f-9fdb-8c9a2adea470 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2bb7c030065d4255a9a8243af1a1e926'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:20.215060 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-f8c11d49-0c72-4a6f-9fdb-8c9a2adea470 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2bb7c030065d4255a9a8243af1a1e926'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:20.222341 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-f8c11d49-0c72-4a6f-9fdb-8c9a2adea470 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2bb7c030065d4255a9a8243af1a1e926'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:20.239307 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1429/2866] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:09:20 2022] DELETE /identity/v3/users/2bb7c030065d4255a9a8243af1a1e926 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:09:20.244147 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07e4fe2-7dfa-4afe-b55d-a186fa5b74a4 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:20.244630 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07e4fe2-7dfa-4afe-b55d-a186fa5b74a4 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:20.244746 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07e4fe2-7dfa-4afe-b55d-a186fa5b74a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:20.263877 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-f07e4fe2-7dfa-4afe-b55d-a186fa5b74a4 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:09:20.265705 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-f07e4fe2-7dfa-4afe-b55d-a186fa5b74a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:20.277871 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:09:20.325248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1438/2867] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:09:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:09:20.335526 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c11d49-0c72-4a6f-9fdb-8c9a2adea470 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:20.336950 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-f8c11d49-0c72-4a6f-9fdb-8c9a2adea470 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:20.394364 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c11d49-0c72-4a6f-9fdb-8c9a2adea470 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:20.397992 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5586cc40-6fa2-4664-ab2a-01e53ba2ca06 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:20.398629 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5586cc40-6fa2-4664-ab2a-01e53ba2ca06 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:20.398901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5586cc40-6fa2-4664-ab2a-01e53ba2ca06 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:20.399177 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5586cc40-6fa2-4664-ab2a-01e53ba2ca06 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:20.408172 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5586cc40-6fa2-4664-ab2a-01e53ba2ca06 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:20.409602 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5586cc40-6fa2-4664-ab2a-01e53ba2ca06 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:20.418958 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1430/2868] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:09:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:09:20.683409 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07e4fe2-7dfa-4afe-b55d-a186fa5b74a4 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:20.684829 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-f07e4fe2-7dfa-4afe-b55d-a186fa5b74a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:20.697209 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07e4fe2-7dfa-4afe-b55d-a186fa5b74a4 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:20.701782 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba5c6d2-5693-4906-b9b7-bd53020401e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:20.702412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba5c6d2-5693-4906-b9b7-bd53020401e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:20.702727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba5c6d2-5693-4906-b9b7-bd53020401e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:20.702999 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba5c6d2-5693-4906-b9b7-bd53020401e3 admin admin] PATH_INFO: `/v3/projects/56ae2847ac8442a587478b17b54953f6` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:20.704119 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba5c6d2-5693-4906-b9b7-bd53020401e3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=56ae2847ac8442a587478b17b54953f6)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:20.706034 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba5c6d2-5693-4906-b9b7-bd53020401e3 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:20.753045 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5ba5c6d2-5693-4906-b9b7-bd53020401e3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '56ae2847ac8442a587478b17b54953f6', 'request_id': 'req-5ba5c6d2-5693-4906-b9b7-bd53020401e3'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:20.758436 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5ba5c6d2-5693-4906-b9b7-bd53020401e3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '56ae2847ac8442a587478b17b54953f6', 'request_id': 'req-5ba5c6d2-5693-4906-b9b7-bd53020401e3'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:20.765522 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5ba5c6d2-5693-4906-b9b7-bd53020401e3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '56ae2847ac8442a587478b17b54953f6', 'request_id': 'req-5ba5c6d2-5693-4906-b9b7-bd53020401e3'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:20.776463 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5ba5c6d2-5693-4906-b9b7-bd53020401e3 admin admin] The token cache is being invalidate because project 56ae2847ac8442a587478b17b54953f6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:09:20.776743 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5ba5c6d2-5693-4906-b9b7-bd53020401e3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:20.777361 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-5ba5c6d2-5693-4906-b9b7-bd53020401e3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:20.778863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1439/2869] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:09:20 2022] DELETE /identity/v3/projects/56ae2847ac8442a587478b17b54953f6 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:09:20.783381 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5586cc40-6fa2-4664-ab2a-01e53ba2ca06 service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:20.792870 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5586cc40-6fa2-4664-ab2a-01e53ba2ca06 service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:20.848031 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51544e46-46a5-4462-890a-cb835f552462 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:20.848658 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51544e46-46a5-4462-890a-cb835f552462 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:20.848933 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51544e46-46a5-4462-890a-cb835f552462 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:20.849204 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51544e46-46a5-4462-890a-cb835f552462 admin admin] PATH_INFO: `/v3/users/b282704274d545ac9d203d6cbabb8207` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:20.850313 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51544e46-46a5-4462-890a-cb835f552462 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b282704274d545ac9d203d6cbabb8207)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:20.852433 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51544e46-46a5-4462-890a-cb835f552462 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:20.895032 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:09:20.897821 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-51544e46-46a5-4462-890a-cb835f552462 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b282704274d545ac9d203d6cbabb8207'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:20.907045 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-51544e46-46a5-4462-890a-cb835f552462 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b282704274d545ac9d203d6cbabb8207'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:20.907660 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-51544e46-46a5-4462-890a-cb835f552462 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b282704274d545ac9d203d6cbabb8207'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:20.908263 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-51544e46-46a5-4462-890a-cb835f552462 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b282704274d545ac9d203d6cbabb8207'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:20.915147 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-51544e46-46a5-4462-890a-cb835f552462 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b282704274d545ac9d203d6cbabb8207'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:20.933202 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1431/2870] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:09:20 2022] DELETE /identity/v3/users/b282704274d545ac9d203d6cbabb8207 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:09:21.202885 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba5c6d2-5693-4906-b9b7-bd53020401e3 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:21.204305 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-5ba5c6d2-5693-4906-b9b7-bd53020401e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:21.224070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba5c6d2-5693-4906-b9b7-bd53020401e3 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:21.280272 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e573b8-ce86-4eb8-b56e-530e9abd62a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:21.281530 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e573b8-ce86-4eb8-b56e-530e9abd62a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:21.281816 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e573b8-ce86-4eb8-b56e-530e9abd62a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:21.282201 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e573b8-ce86-4eb8-b56e-530e9abd62a7 admin admin] PATH_INFO: `/v3/projects/50a05a71f71a490589e659b4290b03b5` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:21.283494 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e573b8-ce86-4eb8-b56e-530e9abd62a7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=50a05a71f71a490589e659b4290b03b5)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:21.285432 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e573b8-ce86-4eb8-b56e-530e9abd62a7 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:21.328549 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:09:21.332326 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d1e573b8-ce86-4eb8-b56e-530e9abd62a7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '50a05a71f71a490589e659b4290b03b5', 'request_id': 'req-d1e573b8-ce86-4eb8-b56e-530e9abd62a7'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:21.339439 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d1e573b8-ce86-4eb8-b56e-530e9abd62a7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '50a05a71f71a490589e659b4290b03b5', 'request_id': 'req-d1e573b8-ce86-4eb8-b56e-530e9abd62a7'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:21.347258 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d1e573b8-ce86-4eb8-b56e-530e9abd62a7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '50a05a71f71a490589e659b4290b03b5', 'request_id': 'req-d1e573b8-ce86-4eb8-b56e-530e9abd62a7'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:21.358282 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d1e573b8-ce86-4eb8-b56e-530e9abd62a7 admin admin] The token cache is being invalidate because project 50a05a71f71a490589e659b4290b03b5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:09:21.358700 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d1e573b8-ce86-4eb8-b56e-530e9abd62a7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:21.359288 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-d1e573b8-ce86-4eb8-b56e-530e9abd62a7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:21.360776 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1440/2871] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:09:21 2022] DELETE /identity/v3/projects/50a05a71f71a490589e659b4290b03b5 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:09:21.366595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7300e7-eeea-43a1-8cf1-22f5c615bc59 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:21.367052 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7300e7-eeea-43a1-8cf1-22f5c615bc59 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:21.367162 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7300e7-eeea-43a1-8cf1-22f5c615bc59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:21.386268 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-4f7300e7-eeea-43a1-8cf1-22f5c615bc59 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:09:21.387488 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-4f7300e7-eeea-43a1-8cf1-22f5c615bc59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:21.445714 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1432/2872] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:09:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:09:21.454531 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a017481-cda2-4ea7-a167-67abb7cf19b5 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:21.454850 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a017481-cda2-4ea7-a167-67abb7cf19b5 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:21.455124 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a017481-cda2-4ea7-a167-67abb7cf19b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:21.475345 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-4a017481-cda2-4ea7-a167-67abb7cf19b5 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:09:21.476550 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-4a017481-cda2-4ea7-a167-67abb7cf19b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:21.484365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1441/2873] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:09:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:09:21.488620 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7300e7-eeea-43a1-8cf1-22f5c615bc59 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:21.498259 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7300e7-eeea-43a1-8cf1-22f5c615bc59 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:21.503430 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c26342-5186-429c-a096-b35ed2f5b4cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:21.504024 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c26342-5186-429c-a096-b35ed2f5b4cb admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:21.504303 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c26342-5186-429c-a096-b35ed2f5b4cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:21.504577 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c26342-5186-429c-a096-b35ed2f5b4cb admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:21.505226 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c26342-5186-429c-a096-b35ed2f5b4cb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:09:21.505529 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c26342-5186-429c-a096-b35ed2f5b4cb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:21.507159 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c26342-5186-429c-a096-b35ed2f5b4cb admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:21.519469 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1433/2874] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:09:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 01:09:21.524394 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a017481-cda2-4ea7-a167-67abb7cf19b5 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:21.533760 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a017481-cda2-4ea7-a167-67abb7cf19b5 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:21.538489 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42540007-7dff-4c05-aa24-88684c40aa1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:21.539087 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42540007-7dff-4c05-aa24-88684c40aa1d admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:21.539365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42540007-7dff-4c05-aa24-88684c40aa1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:21.539636 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42540007-7dff-4c05-aa24-88684c40aa1d admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:21.540262 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42540007-7dff-4c05-aa24-88684c40aa1d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:21.542154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42540007-7dff-4c05-aa24-88684c40aa1d admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:21.570953 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1442/2875] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:09:21 2022] POST /identity/v3/projects => generated 361 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:09:21.575274 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c26342-5186-429c-a096-b35ed2f5b4cb admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:21.585784 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c26342-5186-429c-a096-b35ed2f5b4cb admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:21.646240 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8c0460-3dbf-41af-9f18-5bdd90b03511 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:21.646887 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8c0460-3dbf-41af-9f18-5bdd90b03511 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:21.647162 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8c0460-3dbf-41af-9f18-5bdd90b03511 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:21.647430 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8c0460-3dbf-41af-9f18-5bdd90b03511 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:21.648074 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8c0460-3dbf-41af-9f18-5bdd90b03511 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:21.649986 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8c0460-3dbf-41af-9f18-5bdd90b03511 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:21.670126 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1434/2876] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:09:21 2022] POST /identity/v3/users => generated 281 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:09:21.675857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42540007-7dff-4c05-aa24-88684c40aa1d admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:21.686064 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42540007-7dff-4c05-aa24-88684c40aa1d admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:21.691156 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131e75c8-b296-40a9-85c4-b513c1c15ed5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:21.691745 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131e75c8-b296-40a9-85c4-b513c1c15ed5 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:21.692022 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131e75c8-b296-40a9-85c4-b513c1c15ed5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:21.692296 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131e75c8-b296-40a9-85c4-b513c1c15ed5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:21.692958 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131e75c8-b296-40a9-85c4-b513c1c15ed5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:21.694576 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131e75c8-b296-40a9-85c4-b513c1c15ed5 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:21.705086 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1443/2877] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:09:21 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:09:21.708971 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8c0460-3dbf-41af-9f18-5bdd90b03511 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:21.719160 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8c0460-3dbf-41af-9f18-5bdd90b03511 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:21.723792 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db677ac9-338a-4a3e-b70f-b141572d4ba1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:21.724385 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db677ac9-338a-4a3e-b70f-b141572d4ba1 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:21.724664 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db677ac9-338a-4a3e-b70f-b141572d4ba1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:21.724938 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db677ac9-338a-4a3e-b70f-b141572d4ba1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:21.725578 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db677ac9-338a-4a3e-b70f-b141572d4ba1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:21.727216 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db677ac9-338a-4a3e-b70f-b141572d4ba1 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:21.738880 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1435/2878] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:09:21 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:09:21.743124 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131e75c8-b296-40a9-85c4-b513c1c15ed5 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:21.753450 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131e75c8-b296-40a9-85c4-b513c1c15ed5 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:21.758243 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df529537-656a-4e81-b12f-cb84772595df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:21.758929 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df529537-656a-4e81-b12f-cb84772595df admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:21.759205 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df529537-656a-4e81-b12f-cb84772595df admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:21.759476 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df529537-656a-4e81-b12f-cb84772595df admin admin] PATH_INFO: `/v3/projects/41ab11eb60a447efa09d9a9005dc31a6/users/2e84ccb3a46648519f3df85e91cc259c/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:21.774152 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df529537-656a-4e81-b12f-cb84772595df admin admin] RBAC: Authorizing `identity:create_grant(project_id=41ab11eb60a447efa09d9a9005dc31a6, user_id=2e84ccb3a46648519f3df85e91cc259c, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:21.776812 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df529537-656a-4e81-b12f-cb84772595df admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:21.789552 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1444/2879] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:09:21 2022] PUT /identity/v3/projects/41ab11eb60a447efa09d9a9005dc31a6/users/2e84ccb3a46648519f3df85e91cc259c/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:09:21.853765 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d769f64-c437-41ff-8ad5-1f38475bbf86 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:21.854044 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d769f64-c437-41ff-8ad5-1f38475bbf86 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:21.854312 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d769f64-c437-41ff-8ad5-1f38475bbf86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:21.882559 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-8d769f64-c437-41ff-8ad5-1f38475bbf86 None None] MFA Rules not processed for user `2e84ccb3a46648519f3df85e91cc259c`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:09:21.883767 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8d769f64-c437-41ff-8ad5-1f38475bbf86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:21.896197 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:DELETE=4,keystone:SELECT=92,keystone:INSERT=4 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:09:21.947834 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1436/2880] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:09:21 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:09:21.982625 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df529537-656a-4e81-b12f-cb84772595df admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:21.984903 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-df529537-656a-4e81-b12f-cb84772595df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:22.041743 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df529537-656a-4e81-b12f-cb84772595df admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:22.045653 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037dbdc5-7df9-433d-85bd-661e87d49098 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:22.046863 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037dbdc5-7df9-433d-85bd-661e87d49098 service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:22.047659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037dbdc5-7df9-433d-85bd-661e87d49098 service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:22.047659 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037dbdc5-7df9-433d-85bd-661e87d49098 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:22.057126 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037dbdc5-7df9-433d-85bd-661e87d49098 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:22.058605 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037dbdc5-7df9-433d-85bd-661e87d49098 service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:22.068220 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1445/2881] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:09:21 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:09:22.083594 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d769f64-c437-41ff-8ad5-1f38475bbf86 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:22.144061 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d769f64-c437-41ff-8ad5-1f38475bbf86 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:22.148614 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e946e66-f666-474a-be60-b2a43039fc26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:22.149232 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e946e66-f666-474a-be60-b2a43039fc26 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:22.149510 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e946e66-f666-474a-be60-b2a43039fc26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:22.149781 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e946e66-f666-474a-be60-b2a43039fc26 admin admin] PATH_INFO: `/v3/users/2e84ccb3a46648519f3df85e91cc259c` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:22.150928 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e946e66-f666-474a-be60-b2a43039fc26 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2e84ccb3a46648519f3df85e91cc259c)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:22.152814 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e946e66-f666-474a-be60-b2a43039fc26 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:22.197404 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-2e946e66-f666-474a-be60-b2a43039fc26 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2e84ccb3a46648519f3df85e91cc259c'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:22.205011 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-2e946e66-f666-474a-be60-b2a43039fc26 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2e84ccb3a46648519f3df85e91cc259c'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:22.205696 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-2e946e66-f666-474a-be60-b2a43039fc26 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2e84ccb3a46648519f3df85e91cc259c'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:22.206251 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-2e946e66-f666-474a-be60-b2a43039fc26 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2e84ccb3a46648519f3df85e91cc259c'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:22.213103 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-2e946e66-f666-474a-be60-b2a43039fc26 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2e84ccb3a46648519f3df85e91cc259c'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:22.229814 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1437/2882] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:09:22 2022] DELETE /identity/v3/users/2e84ccb3a46648519f3df85e91cc259c => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:09:22.234765 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d689fb7f-6cba-4382-a234-6a6ac5a28e27 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:22.235048 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d689fb7f-6cba-4382-a234-6a6ac5a28e27 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:22.235319 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d689fb7f-6cba-4382-a234-6a6ac5a28e27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:22.254924 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-d689fb7f-6cba-4382-a234-6a6ac5a28e27 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:09:22.256111 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-d689fb7f-6cba-4382-a234-6a6ac5a28e27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:22.313430 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1446/2883] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:09:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:09:22.323128 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e946e66-f666-474a-be60-b2a43039fc26 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:22.324697 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-2e946e66-f666-474a-be60-b2a43039fc26 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:22.365807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:09:22.384758 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e946e66-f666-474a-be60-b2a43039fc26 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:22.388420 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c6a154-2a9d-44c0-809e-e4ac52d76d7f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:22.389029 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c6a154-2a9d-44c0-809e-e4ac52d76d7f service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:22.389305 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c6a154-2a9d-44c0-809e-e4ac52d76d7f service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:22.389578 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c6a154-2a9d-44c0-809e-e4ac52d76d7f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:22.398639 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c6a154-2a9d-44c0-809e-e4ac52d76d7f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:22.400066 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c6a154-2a9d-44c0-809e-e4ac52d76d7f service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:22.410026 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1438/2884] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:09:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:09:22.675267 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d689fb7f-6cba-4382-a234-6a6ac5a28e27 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:22.676878 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-d689fb7f-6cba-4382-a234-6a6ac5a28e27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:22.690250 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d689fb7f-6cba-4382-a234-6a6ac5a28e27 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:22.695021 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9101e687-90ec-4711-be87-dc4815d78b2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:22.695629 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9101e687-90ec-4711-be87-dc4815d78b2b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:22.696443 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9101e687-90ec-4711-be87-dc4815d78b2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:22.696718 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9101e687-90ec-4711-be87-dc4815d78b2b admin admin] PATH_INFO: `/v3/projects/41ab11eb60a447efa09d9a9005dc31a6` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:22.698007 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9101e687-90ec-4711-be87-dc4815d78b2b admin admin] RBAC: Authorizing `identity:delete_project(project_id=41ab11eb60a447efa09d9a9005dc31a6)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:22.699965 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9101e687-90ec-4711-be87-dc4815d78b2b admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:22.708761 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:DELETE=4,keystone:SELECT=93,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:09:22.748496 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-9101e687-90ec-4711-be87-dc4815d78b2b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '41ab11eb60a447efa09d9a9005dc31a6', 'request_id': 'req-9101e687-90ec-4711-be87-dc4815d78b2b'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:22.754208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-9101e687-90ec-4711-be87-dc4815d78b2b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '41ab11eb60a447efa09d9a9005dc31a6', 'request_id': 'req-9101e687-90ec-4711-be87-dc4815d78b2b'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:22.761523 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-9101e687-90ec-4711-be87-dc4815d78b2b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '41ab11eb60a447efa09d9a9005dc31a6', 'request_id': 'req-9101e687-90ec-4711-be87-dc4815d78b2b'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:22.773311 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-9101e687-90ec-4711-be87-dc4815d78b2b admin admin] The token cache is being invalidate because project 41ab11eb60a447efa09d9a9005dc31a6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:09:22.773519 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-9101e687-90ec-4711-be87-dc4815d78b2b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:22.774164 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-9101e687-90ec-4711-be87-dc4815d78b2b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:09:22.775653 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1447/2885] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:09:22 2022] DELETE /identity/v3/projects/41ab11eb60a447efa09d9a9005dc31a6 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:09:22.781291 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045f7218-9963-4d17-b61a-8d075f17e78d None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:22.781576 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045f7218-9963-4d17-b61a-8d075f17e78d None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:22.781846 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045f7218-9963-4d17-b61a-8d075f17e78d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:22.801207 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-045f7218-9963-4d17-b61a-8d075f17e78d None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:09:22.802405 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-045f7218-9963-4d17-b61a-8d075f17e78d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:22.859727 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1439/2886] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:09:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:09:22.867207 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8f4ec9-67b0-4ffb-a8d2-5d67b3305027 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:22.867500 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8f4ec9-67b0-4ffb-a8d2-5d67b3305027 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:22.867777 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8f4ec9-67b0-4ffb-a8d2-5d67b3305027 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:22.887239 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-6b8f4ec9-67b0-4ffb-a8d2-5d67b3305027 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:09:22.888420 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-6b8f4ec9-67b0-4ffb-a8d2-5d67b3305027 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:22.896983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1448/2887] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:09:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:09:22.901293 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045f7218-9963-4d17-b61a-8d075f17e78d None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:22.912119 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045f7218-9963-4d17-b61a-8d075f17e78d None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:22.917052 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60e41f1-34a9-470d-b2bc-c2b59eec9b60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:22.917658 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60e41f1-34a9-470d-b2bc-c2b59eec9b60 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:22.917951 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60e41f1-34a9-470d-b2bc-c2b59eec9b60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:22.918219 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60e41f1-34a9-470d-b2bc-c2b59eec9b60 admin admin] PATH_INFO: `/v3/domains` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:22.918901 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60e41f1-34a9-470d-b2bc-c2b59eec9b60 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81530) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:09:22.919203 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60e41f1-34a9-470d-b2bc-c2b59eec9b60 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:22.920788 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60e41f1-34a9-470d-b2bc-c2b59eec9b60 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:22.933931 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1440/2888] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:09:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 01:09:22.938175 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8f4ec9-67b0-4ffb-a8d2-5d67b3305027 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:22.948634 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8f4ec9-67b0-4ffb-a8d2-5d67b3305027 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:22.953783 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec90cb5b-358c-4fb2-8401-f71af96b5bd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:22.954401 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec90cb5b-358c-4fb2-8401-f71af96b5bd7 admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:22.954711 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec90cb5b-358c-4fb2-8401-f71af96b5bd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:22.954987 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec90cb5b-358c-4fb2-8401-f71af96b5bd7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:22.955617 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec90cb5b-358c-4fb2-8401-f71af96b5bd7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:22.957508 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec90cb5b-358c-4fb2-8401-f71af96b5bd7 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:22.991660 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1449/2889] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:09:22 2022] POST /identity/v3/projects => generated 371 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:09:22.996104 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60e41f1-34a9-470d-b2bc-c2b59eec9b60 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:23.008781 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60e41f1-34a9-470d-b2bc-c2b59eec9b60 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:23.065481 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77450e25-8ab1-4b87-a019-a988abdf5462 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:23.066100 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77450e25-8ab1-4b87-a019-a988abdf5462 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:23.066430 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77450e25-8ab1-4b87-a019-a988abdf5462 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:23.066737 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77450e25-8ab1-4b87-a019-a988abdf5462 admin admin] PATH_INFO: `/v3/users` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:23.067384 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77450e25-8ab1-4b87-a019-a988abdf5462 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:23.069358 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77450e25-8ab1-4b87-a019-a988abdf5462 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:23.091479 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1441/2890] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:09:22 2022] POST /identity/v3/users => generated 286 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:09:23.096166 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec90cb5b-358c-4fb2-8401-f71af96b5bd7 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:23.107144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec90cb5b-358c-4fb2-8401-f71af96b5bd7 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:23.111811 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617294c5-ead3-499e-b43e-82a12115bb47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:23.112437 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617294c5-ead3-499e-b43e-82a12115bb47 admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:23.112713 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617294c5-ead3-499e-b43e-82a12115bb47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:23.112980 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617294c5-ead3-499e-b43e-82a12115bb47 admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:23.113635 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617294c5-ead3-499e-b43e-82a12115bb47 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:23.115270 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617294c5-ead3-499e-b43e-82a12115bb47 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:23.125545 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1450/2891] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:09:23 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:09:23.129566 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77450e25-8ab1-4b87-a019-a988abdf5462 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:23.139650 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77450e25-8ab1-4b87-a019-a988abdf5462 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:23.144833 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd8b02f-f34f-4eca-ae32-21fce45e5528 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:23.145428 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd8b02f-f34f-4eca-ae32-21fce45e5528 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:23.145758 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd8b02f-f34f-4eca-ae32-21fce45e5528 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:23.146370 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd8b02f-f34f-4eca-ae32-21fce45e5528 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:23.147254 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd8b02f-f34f-4eca-ae32-21fce45e5528 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:23.148834 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd8b02f-f34f-4eca-ae32-21fce45e5528 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:23.159403 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1442/2892] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:09:23 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:09:23.163611 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617294c5-ead3-499e-b43e-82a12115bb47 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:23.173701 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617294c5-ead3-499e-b43e-82a12115bb47 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:23.178178 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4c70b1-7d61-4d4a-bc39-61341650c7c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:23.178841 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4c70b1-7d61-4d4a-bc39-61341650c7c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:23.179117 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4c70b1-7d61-4d4a-bc39-61341650c7c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:23.179390 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4c70b1-7d61-4d4a-bc39-61341650c7c6 admin admin] PATH_INFO: `/v3/projects/c83778305ed1494f931ca66d197fa5d3/users/33ea64ca8ec14dd2a34a05ce90f9fa7a/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:23.195389 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4c70b1-7d61-4d4a-bc39-61341650c7c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c83778305ed1494f931ca66d197fa5d3, user_id=33ea64ca8ec14dd2a34a05ce90f9fa7a, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:23.198831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4c70b1-7d61-4d4a-bc39-61341650c7c6 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:23.212614 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1451/2893] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:09:23 2022] PUT /identity/v3/projects/c83778305ed1494f931ca66d197fa5d3/users/33ea64ca8ec14dd2a34a05ce90f9fa7a/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:09:23.218014 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2428e2eb-fc9e-4458-a3c4-98ffd172d793 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:23.218590 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2428e2eb-fc9e-4458-a3c4-98ffd172d793 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:23.218885 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2428e2eb-fc9e-4458-a3c4-98ffd172d793 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:23.237483 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-2428e2eb-fc9e-4458-a3c4-98ffd172d793 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:09:23.238717 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-2428e2eb-fc9e-4458-a3c4-98ffd172d793 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:23.281134 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:09:23.301943 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1443/2894] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:09:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:09:23.312250 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4c70b1-7d61-4d4a-bc39-61341650c7c6 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:23.313928 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3c4c70b1-7d61-4d4a-bc39-61341650c7c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:23.370829 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4c70b1-7d61-4d4a-bc39-61341650c7c6 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:23.375070 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3936b16-e3ad-41d1-bd66-378e2a6b342e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:23.375678 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3936b16-e3ad-41d1-bd66-378e2a6b342e service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:23.375963 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3936b16-e3ad-41d1-bd66-378e2a6b342e service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:23.376238 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3936b16-e3ad-41d1-bd66-378e2a6b342e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:23.385256 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3936b16-e3ad-41d1-bd66-378e2a6b342e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:23.386729 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3936b16-e3ad-41d1-bd66-378e2a6b342e service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:23.397466 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1452/2895] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:09:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:09:24.017361 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e800da8-dec0-4f75-9ce7-3b8e61f537c6 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:24.017873 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e800da8-dec0-4f75-9ce7-3b8e61f537c6 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:24.018050 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e800da8-dec0-4f75-9ce7-3b8e61f537c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:24.048175 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-8e800da8-dec0-4f75-9ce7-3b8e61f537c6 None None] MFA Rules not processed for user `33ea64ca8ec14dd2a34a05ce90f9fa7a`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:09:24.049553 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-8e800da8-dec0-4f75-9ce7-3b8e61f537c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:24.116320 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1444/2896] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:09:24 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:09:24.127407 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3936b16-e3ad-41d1-bd66-378e2a6b342e service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:24.128831 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c3936b16-e3ad-41d1-bd66-378e2a6b342e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:24.170160 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:09:24.189578 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3936b16-e3ad-41d1-bd66-378e2a6b342e service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:24.193228 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f072eb28-f5a5-4bad-b22a-a78deea2b226 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:24.193840 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f072eb28-f5a5-4bad-b22a-a78deea2b226 service cinder] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:24.194119 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f072eb28-f5a5-4bad-b22a-a78deea2b226 service cinder] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:24.194393 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f072eb28-f5a5-4bad-b22a-a78deea2b226 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:24.203431 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f072eb28-f5a5-4bad-b22a-a78deea2b226 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:24.204857 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f072eb28-f5a5-4bad-b22a-a78deea2b226 service cinder] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:24.213493 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1453/2897] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:09:24 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:09:25.569002 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210ea1ce-9690-461d-bb1b-94e47dfb8c09 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:25.569523 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210ea1ce-9690-461d-bb1b-94e47dfb8c09 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:25.569523 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210ea1ce-9690-461d-bb1b-94e47dfb8c09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:25.588101 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-210ea1ce-9690-461d-bb1b-94e47dfb8c09 None None] MFA Rules not processed for user `33ea64ca8ec14dd2a34a05ce90f9fa7a`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:09:25.589297 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-210ea1ce-9690-461d-bb1b-94e47dfb8c09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:25.597135 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1445/2898] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:09:25 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:09:25.831105 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f072eb28-f5a5-4bad-b22a-a78deea2b226 service cinder] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:25.840542 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f072eb28-f5a5-4bad-b22a-a78deea2b226 service cinder] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:25.844996 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c46626-e753-4a1a-b4c8-ccd387251b7d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:25.845592 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c46626-e753-4a1a-b4c8-ccd387251b7d service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:25.845868 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c46626-e753-4a1a-b4c8-ccd387251b7d service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:25.846138 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c46626-e753-4a1a-b4c8-ccd387251b7d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:25.847882 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-c9c46626-e753-4a1a-b4c8-ccd387251b7d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:25.857290 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c46626-e753-4a1a-b4c8-ccd387251b7d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:25.858752 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c46626-e753-4a1a-b4c8-ccd387251b7d service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:25.866628 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1454/2899] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:09:25 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:09:35.582823 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=42 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:09:35.863142 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=16 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:09:40.876808 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264c2a2e-94e2-439c-bff4-7b5a05341726 None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:40.877422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264c2a2e-94e2-439c-bff4-7b5a05341726 None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:40.877422 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264c2a2e-94e2-439c-bff4-7b5a05341726 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:40.896395 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-264c2a2e-94e2-439c-bff4-7b5a05341726 None None] MFA Rules not processed for user `33ea64ca8ec14dd2a34a05ce90f9fa7a`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:09:40.897608 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-264c2a2e-94e2-439c-bff4-7b5a05341726 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:40.904840 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1446/2900] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:09:40 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:09:41.146489 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c46626-e753-4a1a-b4c8-ccd387251b7d service neutron] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:09:41.160680 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c46626-e753-4a1a-b4c8-ccd387251b7d service neutron] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:09:41.164930 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfe08d2-9959-4a4f-8e69-6ac92301564d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:09:41.165537 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfe08d2-9959-4a4f-8e69-6ac92301564d service neutron] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:09:41.165950 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfe08d2-9959-4a4f-8e69-6ac92301564d service neutron] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:09:41.166359 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfe08d2-9959-4a4f-8e69-6ac92301564d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:09:41.168248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-abfe08d2-9959-4a4f-8e69-6ac92301564d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:09:41.178591 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfe08d2-9959-4a4f-8e69-6ac92301564d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:09:41.180002 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfe08d2-9959-4a4f-8e69-6ac92301564d service neutron] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:09:41.188043 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1455/2901] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:09:41 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:09:50.890892 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=3 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:09:51.185068 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=6 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:10:00.485466 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264c2a2e-94e2-439c-bff4-7b5a05341726 None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:10:00.487040 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-264c2a2e-94e2-439c-bff4-7b5a05341726 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:10:00.610148 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264c2a2e-94e2-439c-bff4-7b5a05341726 None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:10:00.617276 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccefd96b-e923-4ae4-8784-6aae0dabb192 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8f30116fb5ce43dab69efa8bafd9d049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:10:00.617995 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccefd96b-e923-4ae4-8784-6aae0dabb192 service placement] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:00.618286 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccefd96b-e923-4ae4-8784-6aae0dabb192 service placement] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:00.618595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccefd96b-e923-4ae4-8784-6aae0dabb192 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:00.620504 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ccefd96b-e923-4ae4-8784-6aae0dabb192 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:10:00.634037 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccefd96b-e923-4ae4-8784-6aae0dabb192 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:10:00.636308 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccefd96b-e923-4ae4-8784-6aae0dabb192 service placement] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:10:00.648065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1447/2902] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:10:00 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:10:01.024869 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f98da9b-e539-46c3-b873-248a8e25b2c7 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:01.025412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f98da9b-e539-46c3-b873-248a8e25b2c7 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:01.025412 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f98da9b-e539-46c3-b873-248a8e25b2c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:01.077116 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-5f98da9b-e539-46c3-b873-248a8e25b2c7 None None] MFA Rules not processed for user `3e31dce69c344c9aa4a55f31b9c735df`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:10:01.078365 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-5f98da9b-e539-46c3-b873-248a8e25b2c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:10:01.151402 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1456/2903] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 18 01:10:01 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:10:01.202544 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccefd96b-e923-4ae4-8784-6aae0dabb192 service placement] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:10:01.204328 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-ccefd96b-e923-4ae4-8784-6aae0dabb192 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:10:01.300022 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccefd96b-e923-4ae4-8784-6aae0dabb192 service placement] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:10:01.319053 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94a175b-4263-4ea0-9654-c02c08aa2f7f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6230ed7ce0eb43e9b2b8530a28eb6299', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:10:01.319676 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94a175b-4263-4ea0-9654-c02c08aa2f7f service swift] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:01.319950 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94a175b-4263-4ea0-9654-c02c08aa2f7f service swift] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:01.320218 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94a175b-4263-4ea0-9654-c02c08aa2f7f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:01.329013 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94a175b-4263-4ea0-9654-c02c08aa2f7f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:10:01.330616 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94a175b-4263-4ea0-9654-c02c08aa2f7f service swift] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:10:01.339895 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1448/2904] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 18 01:10:01 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 18 01:10:04.396535 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f98da9b-e539-46c3-b873-248a8e25b2c7 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:10:04.398017 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-5f98da9b-e539-46c3-b873-248a8e25b2c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:10:04.413410 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f98da9b-e539-46c3-b873-248a8e25b2c7 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:10:04.428651 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9fe2e8-c20c-4d99-ba61-82b9eb7a3ff8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:10:04.429197 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9fe2e8-c20c-4d99-ba61-82b9eb7a3ff8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:04.429555 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9fe2e8-c20c-4d99-ba61-82b9eb7a3ff8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:04.429889 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9fe2e8-c20c-4d99-ba61-82b9eb7a3ff8 admin admin] PATH_INFO: `/v3/users/7ea84c1e0dcf441fb1346eaaef4de421` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:04.431242 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9fe2e8-c20c-4d99-ba61-82b9eb7a3ff8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7ea84c1e0dcf441fb1346eaaef4de421)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:10:04.434249 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9fe2e8-c20c-4d99-ba61-82b9eb7a3ff8 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:10:04.498573 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-3a9fe2e8-c20c-4d99-ba61-82b9eb7a3ff8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7ea84c1e0dcf441fb1346eaaef4de421'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:04.506896 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-3a9fe2e8-c20c-4d99-ba61-82b9eb7a3ff8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7ea84c1e0dcf441fb1346eaaef4de421'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:04.529286 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-3a9fe2e8-c20c-4d99-ba61-82b9eb7a3ff8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7ea84c1e0dcf441fb1346eaaef4de421'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:04.530089 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-3a9fe2e8-c20c-4d99-ba61-82b9eb7a3ff8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7ea84c1e0dcf441fb1346eaaef4de421'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:04.544949 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-3a9fe2e8-c20c-4d99-ba61-82b9eb7a3ff8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7ea84c1e0dcf441fb1346eaaef4de421'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:04.552756 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1457/2905] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:10:04 2022] DELETE /identity/v3/users/7ea84c1e0dcf441fb1346eaaef4de421 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 01:10:04.761688 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94a175b-4263-4ea0-9654-c02c08aa2f7f service swift] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:10:04.779265 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94a175b-4263-4ea0-9654-c02c08aa2f7f service swift] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:10:04.857563 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fbac9c-1837-4e80-9a0d-691d4ddd117e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:10:04.858580 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fbac9c-1837-4e80-9a0d-691d4ddd117e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:04.859057 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fbac9c-1837-4e80-9a0d-691d4ddd117e admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:04.859499 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fbac9c-1837-4e80-9a0d-691d4ddd117e admin admin] PATH_INFO: `/v3/projects/2d812ad5fce24a49b0f2d07081d5718a` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:04.860844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88fbac9c-1837-4e80-9a0d-691d4ddd117e admin admin] RBAC: Authorizing `identity:delete_project(project_id=2d812ad5fce24a49b0f2d07081d5718a)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:10:04.863009 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88fbac9c-1837-4e80-9a0d-691d4ddd117e admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:10:04.865098 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=100 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:10:04.923139 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-88fbac9c-1837-4e80-9a0d-691d4ddd117e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2d812ad5fce24a49b0f2d07081d5718a', 'request_id': 'req-88fbac9c-1837-4e80-9a0d-691d4ddd117e'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:04.933175 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-88fbac9c-1837-4e80-9a0d-691d4ddd117e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2d812ad5fce24a49b0f2d07081d5718a', 'request_id': 'req-88fbac9c-1837-4e80-9a0d-691d4ddd117e'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:04.942354 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-88fbac9c-1837-4e80-9a0d-691d4ddd117e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2d812ad5fce24a49b0f2d07081d5718a', 'request_id': 'req-88fbac9c-1837-4e80-9a0d-691d4ddd117e'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:04.955014 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-88fbac9c-1837-4e80-9a0d-691d4ddd117e admin admin] The token cache is being invalidate because project 2d812ad5fce24a49b0f2d07081d5718a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:10:04.955306 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-88fbac9c-1837-4e80-9a0d-691d4ddd117e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:04.955961 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-88fbac9c-1837-4e80-9a0d-691d4ddd117e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:04.957486 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1449/2906] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:10:04 2022] DELETE /identity/v3/projects/2d812ad5fce24a49b0f2d07081d5718a => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:10:04.961456 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9fe2e8-c20c-4d99-ba61-82b9eb7a3ff8 admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:10:04.963261 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3a9fe2e8-c20c-4d99-ba61-82b9eb7a3ff8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:10:04.978810 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9fe2e8-c20c-4d99-ba61-82b9eb7a3ff8 admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:10:05.030855 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:10:05.052067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748746f2-e154-449c-b3fb-fef2fe087813 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:10:05.053200 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748746f2-e154-449c-b3fb-fef2fe087813 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:05.053727 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748746f2-e154-449c-b3fb-fef2fe087813 admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:05.054219 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748746f2-e154-449c-b3fb-fef2fe087813 admin admin] PATH_INFO: `/v3/users/b19a882b58b14abda11e33e416356b99` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:05.055612 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748746f2-e154-449c-b3fb-fef2fe087813 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b19a882b58b14abda11e33e416356b99)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:10:05.057580 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748746f2-e154-449c-b3fb-fef2fe087813 admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:10:05.117896 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-748746f2-e154-449c-b3fb-fef2fe087813 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b19a882b58b14abda11e33e416356b99'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:05.126629 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-748746f2-e154-449c-b3fb-fef2fe087813 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b19a882b58b14abda11e33e416356b99'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:05.140284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-748746f2-e154-449c-b3fb-fef2fe087813 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b19a882b58b14abda11e33e416356b99'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:05.141266 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-748746f2-e154-449c-b3fb-fef2fe087813 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b19a882b58b14abda11e33e416356b99'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:05.149737 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-748746f2-e154-449c-b3fb-fef2fe087813 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b19a882b58b14abda11e33e416356b99'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:05.157498 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1458/2907] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:10:04 2022] DELETE /identity/v3/users/b19a882b58b14abda11e33e416356b99 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:10:05.394966 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fbac9c-1837-4e80-9a0d-691d4ddd117e admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:10:05.396914 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-88fbac9c-1837-4e80-9a0d-691d4ddd117e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:10:05.420409 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fbac9c-1837-4e80-9a0d-691d4ddd117e admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:10:05.500142 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f397ea6-4865-42ff-a933-d242bc6ae51e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:10:05.500917 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f397ea6-4865-42ff-a933-d242bc6ae51e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:05.501265 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f397ea6-4865-42ff-a933-d242bc6ae51e admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:05.501607 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f397ea6-4865-42ff-a933-d242bc6ae51e admin admin] PATH_INFO: `/v3/projects/ba1f93ecb12540f5bb7f7e0e958776e5` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:05.502849 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f397ea6-4865-42ff-a933-d242bc6ae51e admin admin] RBAC: Authorizing `identity:delete_project(project_id=ba1f93ecb12540f5bb7f7e0e958776e5)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:10:05.504871 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f397ea6-4865-42ff-a933-d242bc6ae51e admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:10:05.563364 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9f397ea6-4865-42ff-a933-d242bc6ae51e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ba1f93ecb12540f5bb7f7e0e958776e5', 'request_id': 'req-9f397ea6-4865-42ff-a933-d242bc6ae51e'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:05.571331 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9f397ea6-4865-42ff-a933-d242bc6ae51e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ba1f93ecb12540f5bb7f7e0e958776e5', 'request_id': 'req-9f397ea6-4865-42ff-a933-d242bc6ae51e'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:05.582120 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9f397ea6-4865-42ff-a933-d242bc6ae51e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ba1f93ecb12540f5bb7f7e0e958776e5', 'request_id': 'req-9f397ea6-4865-42ff-a933-d242bc6ae51e'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:05.596766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9f397ea6-4865-42ff-a933-d242bc6ae51e admin admin] The token cache is being invalidate because project ba1f93ecb12540f5bb7f7e0e958776e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:10:05.596766 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9f397ea6-4865-42ff-a933-d242bc6ae51e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:05.598277 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-9f397ea6-4865-42ff-a933-d242bc6ae51e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:05.599844 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1450/2908] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:10:05 2022] DELETE /identity/v3/projects/ba1f93ecb12540f5bb7f7e0e958776e5 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:10:15.152882 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=33,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:10:15.591827 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=75,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:10:16.741760 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d78424-3e21-4324-9d0e-625093496337 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:16.742274 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d78424-3e21-4324-9d0e-625093496337 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:16.742380 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d78424-3e21-4324-9d0e-625093496337 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:16.761292 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-22d78424-3e21-4324-9d0e-625093496337 None None] MFA Rules not processed for user `3e31dce69c344c9aa4a55f31b9c735df`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:10:16.762512 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-22d78424-3e21-4324-9d0e-625093496337 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:10:16.815366 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1459/2909] 10.222.0.42 () {58 vars in 967 bytes} [Sun Sep 18 01:10:16 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:10:16.827425 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f397ea6-4865-42ff-a933-d242bc6ae51e admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:10:16.828815 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-9f397ea6-4865-42ff-a933-d242bc6ae51e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:10:16.897227 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f397ea6-4865-42ff-a933-d242bc6ae51e admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:10:16.902477 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34949615-b570-4b2d-a296-f1172c107d82 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6230ed7ce0eb43e9b2b8530a28eb6299', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:10:16.903099 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34949615-b570-4b2d-a296-f1172c107d82 service swift] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:16.903376 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34949615-b570-4b2d-a296-f1172c107d82 service swift] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:16.903646 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34949615-b570-4b2d-a296-f1172c107d82 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:16.912252 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34949615-b570-4b2d-a296-f1172c107d82 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:10:16.913807 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34949615-b570-4b2d-a296-f1172c107d82 service swift] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:10:16.921651 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1451/2910] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 18 01:10:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 18 01:10:18.841925 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d78424-3e21-4324-9d0e-625093496337 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:10:18.843430 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-22d78424-3e21-4324-9d0e-625093496337 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:10:18.910434 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d78424-3e21-4324-9d0e-625093496337 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:10:18.915000 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2af77ca-d2b5-4c12-b4ad-c11bafb29a5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:10:18.915626 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2af77ca-d2b5-4c12-b4ad-c11bafb29a5d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:18.915907 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2af77ca-d2b5-4c12-b4ad-c11bafb29a5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:18.916180 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2af77ca-d2b5-4c12-b4ad-c11bafb29a5d admin admin] PATH_INFO: `/v3/users/33ea64ca8ec14dd2a34a05ce90f9fa7a` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:18.917310 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2af77ca-d2b5-4c12-b4ad-c11bafb29a5d admin admin] RBAC: Authorizing `identity:delete_user(user_id=33ea64ca8ec14dd2a34a05ce90f9fa7a)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:10:18.919277 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2af77ca-d2b5-4c12-b4ad-c11bafb29a5d admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:10:18.965425 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-e2af77ca-d2b5-4c12-b4ad-c11bafb29a5d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '33ea64ca8ec14dd2a34a05ce90f9fa7a'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:18.972810 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-e2af77ca-d2b5-4c12-b4ad-c11bafb29a5d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '33ea64ca8ec14dd2a34a05ce90f9fa7a'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:18.984848 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-e2af77ca-d2b5-4c12-b4ad-c11bafb29a5d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '33ea64ca8ec14dd2a34a05ce90f9fa7a'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:18.985493 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-e2af77ca-d2b5-4c12-b4ad-c11bafb29a5d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '33ea64ca8ec14dd2a34a05ce90f9fa7a'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:18.992591 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-e2af77ca-d2b5-4c12-b4ad-c11bafb29a5d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '33ea64ca8ec14dd2a34a05ce90f9fa7a'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:18.998394 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1460/2911] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:10:18 2022] DELETE /identity/v3/users/33ea64ca8ec14dd2a34a05ce90f9fa7a => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:10:19.168780 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34949615-b570-4b2d-a296-f1172c107d82 service swift] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:10:19.181330 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34949615-b570-4b2d-a296-f1172c107d82 service swift] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:10:19.239769 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e368a7-2729-4533-b982-cf7188ec4e7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:10:19.240401 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e368a7-2729-4533-b982-cf7188ec4e7a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:19.240677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e368a7-2729-4533-b982-cf7188ec4e7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:19.240957 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e368a7-2729-4533-b982-cf7188ec4e7a admin admin] PATH_INFO: `/v3/projects/c83778305ed1494f931ca66d197fa5d3` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:19.242083 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e368a7-2729-4533-b982-cf7188ec4e7a admin admin] RBAC: Authorizing `identity:delete_project(project_id=c83778305ed1494f931ca66d197fa5d3)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:10:19.244030 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e368a7-2729-4533-b982-cf7188ec4e7a admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:10:19.290661 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-84e368a7-2729-4533-b982-cf7188ec4e7a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c83778305ed1494f931ca66d197fa5d3', 'request_id': 'req-84e368a7-2729-4533-b982-cf7188ec4e7a'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:19.296056 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-84e368a7-2729-4533-b982-cf7188ec4e7a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c83778305ed1494f931ca66d197fa5d3', 'request_id': 'req-84e368a7-2729-4533-b982-cf7188ec4e7a'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:19.303342 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-84e368a7-2729-4533-b982-cf7188ec4e7a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c83778305ed1494f931ca66d197fa5d3', 'request_id': 'req-84e368a7-2729-4533-b982-cf7188ec4e7a'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:19.314735 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-84e368a7-2729-4533-b982-cf7188ec4e7a admin admin] The token cache is being invalidate because project c83778305ed1494f931ca66d197fa5d3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81530) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:10:19.315021 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-84e368a7-2729-4533-b982-cf7188ec4e7a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:19.315652 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-84e368a7-2729-4533-b982-cf7188ec4e7a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:10:19.317617 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1452/2912] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:10:19 2022] DELETE /identity/v3/projects/c83778305ed1494f931ca66d197fa5d3 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:10:19.322615 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32007a1e-3530-43df-a8fd-06de669be3d5 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:19.322904 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32007a1e-3530-43df-a8fd-06de669be3d5 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:19.323181 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32007a1e-3530-43df-a8fd-06de669be3d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:19.343067 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-32007a1e-3530-43df-a8fd-06de669be3d5 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:10:19.344267 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-32007a1e-3530-43df-a8fd-06de669be3d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:10:19.362957 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:10:19.404438 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1461/2913] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:10:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:10:19.412208 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ef8d03-8efa-4582-9f11-6bc6cbf06a3a None None] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:19.412426 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ef8d03-8efa-4582-9f11-6bc6cbf06a3a None None] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:19.412721 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ef8d03-8efa-4582-9f11-6bc6cbf06a3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:19.432133 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.auth.core [None req-54ef8d03-8efa-4582-9f11-6bc6cbf06a3a None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81530) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:10:19.433325 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-54ef8d03-8efa-4582-9f11-6bc6cbf06a3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:10:19.441677 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1453/2914] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:10:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:10:19.445898 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32007a1e-3530-43df-a8fd-06de669be3d5 None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:10:19.456023 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32007a1e-3530-43df-a8fd-06de669be3d5 None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:10:19.462658 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e638a2-d3b4-4358-9f0c-81ac8590534b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:10:19.463539 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e638a2-d3b4-4358-9f0c-81ac8590534b admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:19.463983 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e638a2-d3b4-4358-9f0c-81ac8590534b admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:19.464391 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e638a2-d3b4-4358-9f0c-81ac8590534b admin admin] PATH_INFO: `/v3/domains` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:19.465208 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e638a2-d3b4-4358-9f0c-81ac8590534b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81529) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 18 01:10:19.465527 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e638a2-d3b4-4358-9f0c-81ac8590534b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:10:19.467276 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e638a2-d3b4-4358-9f0c-81ac8590534b admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:10:19.479336 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1462/2915] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 18 01:10:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 18 01:10:19.483769 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ef8d03-8efa-4582-9f11-6bc6cbf06a3a None None] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:10:19.493364 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ef8d03-8efa-4582-9f11-6bc6cbf06a3a None None] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:10:19.498035 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce97cf8-133a-43ef-b7dd-0aa2a13aaeb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:10:19.498653 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce97cf8-133a-43ef-b7dd-0aa2a13aaeb7 admin admin] REQUEST_METHOD: `POST` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:19.498929 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce97cf8-133a-43ef-b7dd-0aa2a13aaeb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:19.499195 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce97cf8-133a-43ef-b7dd-0aa2a13aaeb7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:19.499824 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce97cf8-133a-43ef-b7dd-0aa2a13aaeb7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:10:19.501914 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce97cf8-133a-43ef-b7dd-0aa2a13aaeb7 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:10:19.513817 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:10:19.535496 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1454/2916] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 18 01:10:19 2022] POST /identity/v3/projects => generated 383 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 18 01:10:19.539819 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e638a2-d3b4-4358-9f0c-81ac8590534b admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:10:19.549613 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e638a2-d3b4-4358-9f0c-81ac8590534b admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:10:19.605479 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef145e10-5eb9-4f33-a3ab-c1a4f619309f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:10:19.606105 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef145e10-5eb9-4f33-a3ab-c1a4f619309f admin admin] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:19.606381 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef145e10-5eb9-4f33-a3ab-c1a4f619309f admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:19.606688 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef145e10-5eb9-4f33-a3ab-c1a4f619309f admin admin] PATH_INFO: `/v3/users` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:19.607322 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef145e10-5eb9-4f33-a3ab-c1a4f619309f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:10:19.609302 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef145e10-5eb9-4f33-a3ab-c1a4f619309f admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:10:19.630145 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1463/2917] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 18 01:10:19 2022] POST /identity/v3/users => generated 292 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 18 01:10:19.634370 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce97cf8-133a-43ef-b7dd-0aa2a13aaeb7 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:10:19.644530 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce97cf8-133a-43ef-b7dd-0aa2a13aaeb7 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:10:19.649059 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866f257e-3395-461b-915c-1d3a658a22a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:10:19.649649 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866f257e-3395-461b-915c-1d3a658a22a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:19.649922 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866f257e-3395-461b-915c-1d3a658a22a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:19.650190 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866f257e-3395-461b-915c-1d3a658a22a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:19.650863 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866f257e-3395-461b-915c-1d3a658a22a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:10:19.652439 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866f257e-3395-461b-915c-1d3a658a22a7 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:10:19.662089 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1455/2918] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:10:19 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:10:19.666020 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef145e10-5eb9-4f33-a3ab-c1a4f619309f admin admin] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:10:19.676458 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef145e10-5eb9-4f33-a3ab-c1a4f619309f admin admin] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:10:19.681122 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18be2e95-2c7a-4239-8f4a-dbe02cdbb70d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:10:19.681717 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18be2e95-2c7a-4239-8f4a-dbe02cdbb70d admin admin] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:19.681998 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18be2e95-2c7a-4239-8f4a-dbe02cdbb70d admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:19.682271 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18be2e95-2c7a-4239-8f4a-dbe02cdbb70d admin admin] PATH_INFO: `/v3/roles` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:19.682954 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18be2e95-2c7a-4239-8f4a-dbe02cdbb70d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:10:19.684557 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18be2e95-2c7a-4239-8f4a-dbe02cdbb70d admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:10:19.695710 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1464/2919] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 18 01:10:19 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 18 01:10:19.700440 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866f257e-3395-461b-915c-1d3a658a22a7 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:10:19.710206 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866f257e-3395-461b-915c-1d3a658a22a7 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:10:19.714857 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e0dc39-6dac-4403-bc40-12558e7fd564 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:10:19.715495 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e0dc39-6dac-4403-bc40-12558e7fd564 admin admin] REQUEST_METHOD: `PUT` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:19.715770 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e0dc39-6dac-4403-bc40-12558e7fd564 admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:19.716041 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e0dc39-6dac-4403-bc40-12558e7fd564 admin admin] PATH_INFO: `/v3/projects/a4f90e9a8c164448a214481704ff18a2/users/cbfab0fe9b4f4840a3ad9953d90e86bb/roles/18f6cee478e54567b70d5c7557fe694e` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:19.731420 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e0dc39-6dac-4403-bc40-12558e7fd564 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a4f90e9a8c164448a214481704ff18a2, user_id=cbfab0fe9b4f4840a3ad9953d90e86bb, role_id=18f6cee478e54567b70d5c7557fe694e)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:10:19.734046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e0dc39-6dac-4403-bc40-12558e7fd564 admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:10:19.747407 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1456/2920] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 18 01:10:19 2022] PUT /identity/v3/projects/a4f90e9a8c164448a214481704ff18a2/users/cbfab0fe9b4f4840a3ad9953d90e86bb/roles/18f6cee478e54567b70d5c7557fe694e => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:10:19.752559 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ae807c-e625-4edc-807f-33d074dc5467 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:19.752844 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ae807c-e625-4edc-807f-33d074dc5467 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:19.753130 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ae807c-e625-4edc-807f-33d074dc5467 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:19.772154 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-72ae807c-e625-4edc-807f-33d074dc5467 None None] MFA Rules not processed for user `6f5558a706da458791737ed054e51874`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:10:19.773350 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-72ae807c-e625-4edc-807f-33d074dc5467 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:10:19.830707 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1465/2921] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:10:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 18 01:10:19.840434 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e0dc39-6dac-4403-bc40-12558e7fd564 admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:10:19.842204 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-04e0dc39-6dac-4403-bc40-12558e7fd564 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:10:19.899334 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e0dc39-6dac-4403-bc40-12558e7fd564 admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:10:19.902984 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9d1325-d129-4dd7-9850-ffff53b4e7d6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9b0510812b60454586a781e6399e77a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:10:19.903585 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9d1325-d129-4dd7-9850-ffff53b4e7d6 service neutron] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:19.903860 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9d1325-d129-4dd7-9850-ffff53b4e7d6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:19.904131 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9d1325-d129-4dd7-9850-ffff53b4e7d6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:19.912892 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa9d1325-d129-4dd7-9850-ffff53b4e7d6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:10:19.914335 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa9d1325-d129-4dd7-9850-ffff53b4e7d6 service neutron] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:10:19.923733 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1457/2922] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:10:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:10:20.557832 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3193dac2-cc0e-49de-8541-62580ed8294f None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:20.558387 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3193dac2-cc0e-49de-8541-62580ed8294f None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:20.558656 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3193dac2-cc0e-49de-8541-62580ed8294f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:20.587123 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-3193dac2-cc0e-49de-8541-62580ed8294f None None] MFA Rules not processed for user `cbfab0fe9b4f4840a3ad9953d90e86bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:10:20.588346 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-3193dac2-cc0e-49de-8541-62580ed8294f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:10:20.608284 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:10:20.662527 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1466/2923] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:10:20 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:10:20.673217 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9d1325-d129-4dd7-9850-ffff53b4e7d6 service neutron] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:10:20.674621 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-aa9d1325-d129-4dd7-9850-ffff53b4e7d6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:10:20.733608 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9d1325-d129-4dd7-9850-ffff53b4e7d6 service neutron] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:10:20.737890 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68bec86-9259-4140-8b80-4a2480bbeaf9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:10:20.738595 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68bec86-9259-4140-8b80-4a2480bbeaf9 service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:20.738872 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68bec86-9259-4140-8b80-4a2480bbeaf9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:20.739141 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68bec86-9259-4140-8b80-4a2480bbeaf9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:20.748381 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68bec86-9259-4140-8b80-4a2480bbeaf9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:10:20.749794 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68bec86-9259-4140-8b80-4a2480bbeaf9 service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:10:20.759065 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1458/2924] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:10:20 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:10:22.116298 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966063b6-284d-42a6-8203-a410749662b6 None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:10:22.116849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966063b6-284d-42a6-8203-a410749662b6 None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:10:22.116849 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966063b6-284d-42a6-8203-a410749662b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:10:22.135677 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-966063b6-284d-42a6-8203-a410749662b6 None None] MFA Rules not processed for user `cbfab0fe9b4f4840a3ad9953d90e86bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:10:22.136893 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-966063b6-284d-42a6-8203-a410749662b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:10:22.143584 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1467/2925] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:10:22 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:10:30.759059 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=70,keystone:INSERT=2 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:10:32.130248 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=21 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:11:02.040466 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68bec86-9259-4140-8b80-4a2480bbeaf9 service cinder] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:11:02.041839 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-b68bec86-9259-4140-8b80-4a2480bbeaf9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:11:02.110187 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68bec86-9259-4140-8b80-4a2480bbeaf9 service cinder] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:11:02.115200 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5cd851-6b0a-4dd4-a842-f1197f20fa73 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8f30116fb5ce43dab69efa8bafd9d049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:11:02.115801 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5cd851-6b0a-4dd4-a842-f1197f20fa73 service placement] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:11:02.116078 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5cd851-6b0a-4dd4-a842-f1197f20fa73 service placement] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:11:02.116349 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5cd851-6b0a-4dd4-a842-f1197f20fa73 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:11:02.118013 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-0c5cd851-6b0a-4dd4-a842-f1197f20fa73 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:11:02.129131 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5cd851-6b0a-4dd4-a842-f1197f20fa73 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:11:02.131207 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5cd851-6b0a-4dd4-a842-f1197f20fa73 service placement] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:11:02.140764 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1459/2926] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:11:02 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:11:12.136606 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=32 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:11:23.211521 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1835cc41-9047-40a1-80a0-437d88aaf13b None None] REQUEST_METHOD: `POST` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:11:23.212028 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1835cc41-9047-40a1-80a0-437d88aaf13b None None] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:11:23.212144 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1835cc41-9047-40a1-80a0-437d88aaf13b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:11:23.230928 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.auth.core [None req-1835cc41-9047-40a1-80a0-437d88aaf13b None None] MFA Rules not processed for user `cbfab0fe9b4f4840a3ad9953d90e86bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81529) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 18 01:11:23.232118 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-1835cc41-9047-40a1-80a0-437d88aaf13b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:11:23.238876 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1468/2927] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 18 01:11:23 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 18 01:11:33.225603 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=3 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:11:50.773000 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5cd851-6b0a-4dd4-a842-f1197f20fa73 service placement] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:11:50.782070 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5cd851-6b0a-4dd4-a842-f1197f20fa73 service placement] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:11:50.785654 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a3d830-be8d-4a81-9409-654d75709a6f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c4c3eaee0a8f4de0b79841a95b35091f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:11:50.786237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a3d830-be8d-4a81-9409-654d75709a6f service cinder] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:11:50.786543 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a3d830-be8d-4a81-9409-654d75709a6f service cinder] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:11:50.786815 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a3d830-be8d-4a81-9409-654d75709a6f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:11:50.788490 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-13a3d830-be8d-4a81-9409-654d75709a6f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:11:50.856814 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a3d830-be8d-4a81-9409-654d75709a6f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:11:50.858237 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a3d830-be8d-4a81-9409-654d75709a6f service cinder] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:11:50.869989 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1460/2928] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:11:50 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:12:00.864147 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=34 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:12:18.404289 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1835cc41-9047-40a1-80a0-437d88aaf13b None None] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:12:18.405737 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-1835cc41-9047-40a1-80a0-437d88aaf13b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:12:18.420974 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1835cc41-9047-40a1-80a0-437d88aaf13b None None] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:12:18.426015 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc272470-da53-4b41-930e-51d1bacee829 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8f30116fb5ce43dab69efa8bafd9d049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:12:18.427069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc272470-da53-4b41-930e-51d1bacee829 service placement] REQUEST_METHOD: `GET` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:12:18.427069 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc272470-da53-4b41-930e-51d1bacee829 service placement] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:12:18.427318 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc272470-da53-4b41-930e-51d1bacee829 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:12:18.428878 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.fernet_utils [None req-cc272470-da53-4b41-930e-51d1bacee829 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81529) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:12:18.441949 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc272470-da53-4b41-930e-51d1bacee829 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:12:18.443761 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc272470-da53-4b41-930e-51d1bacee829 service placement] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:12:18.460377 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1469/2929] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:12:18 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:12:18.460624 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a3d830-be8d-4a81-9409-654d75709a6f service cinder] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:12:18.472302 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a3d830-be8d-4a81-9409-654d75709a6f service cinder] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:12:18.477533 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89304bfb-eeae-449a-ab50-0dd072cd326d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:12:18.478174 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89304bfb-eeae-449a-ab50-0dd072cd326d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:12:18.479046 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89304bfb-eeae-449a-ab50-0dd072cd326d admin admin] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:12:18.479331 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89304bfb-eeae-449a-ab50-0dd072cd326d admin admin] PATH_INFO: `/v3/users/cbfab0fe9b4f4840a3ad9953d90e86bb` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:12:18.480490 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89304bfb-eeae-449a-ab50-0dd072cd326d admin admin] RBAC: Authorizing `identity:delete_user(user_id=cbfab0fe9b4f4840a3ad9953d90e86bb)` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:12:18.483411 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89304bfb-eeae-449a-ab50-0dd072cd326d admin admin] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:12:18.537505 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-89304bfb-eeae-449a-ab50-0dd072cd326d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cbfab0fe9b4f4840a3ad9953d90e86bb'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:12:18.544759 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-89304bfb-eeae-449a-ab50-0dd072cd326d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cbfab0fe9b4f4840a3ad9953d90e86bb'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:12:18.545461 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-89304bfb-eeae-449a-ab50-0dd072cd326d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cbfab0fe9b4f4840a3ad9953d90e86bb'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:12:18.546066 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-89304bfb-eeae-449a-ab50-0dd072cd326d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cbfab0fe9b4f4840a3ad9953d90e86bb'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:12:18.552899 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.notifications [None req-89304bfb-eeae-449a-ab50-0dd072cd326d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cbfab0fe9b4f4840a3ad9953d90e86bb'} {{(pid=81530) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:12:18.569840 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1461/2930] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 18 01:12:18 2022] DELETE /identity/v3/users/cbfab0fe9b4f4840a3ad9953d90e86bb => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 18 01:12:18.740480 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc272470-da53-4b41-930e-51d1bacee829 service placement] Authenticating user token {{(pid=81529) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:12:18.755807 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc272470-da53-4b41-930e-51d1bacee829 service placement] Validating token access rules against request {{(pid=81529) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:12:18.813047 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0dc5bdd-d9ef-4e58-a620-3c75ce3687cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6f5558a706da458791737ed054e51874', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e41e0644d514666af8ca7ed05ebaec8', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81529) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:12:18.813673 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dc5bdd-d9ef-4e58-a620-3c75ce3687cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:12:18.813950 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dc5bdd-d9ef-4e58-a620-3c75ce3687cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:12:18.814223 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dc5bdd-d9ef-4e58-a620-3c75ce3687cd admin admin] PATH_INFO: `/v3/projects/a4f90e9a8c164448a214481704ff18a2` {{(pid=81529) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:12:18.815372 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0dc5bdd-d9ef-4e58-a620-3c75ce3687cd admin admin] RBAC: Authorizing `identity:delete_project(project_id=a4f90e9a8c164448a214481704ff18a2)` {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:12:18.817260 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0dc5bdd-d9ef-4e58-a620-3c75ce3687cd admin admin] RBAC: Authorization granted {{(pid=81529) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:12:18.862560 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a0dc5bdd-d9ef-4e58-a620-3c75ce3687cd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a4f90e9a8c164448a214481704ff18a2', 'request_id': 'req-a0dc5bdd-d9ef-4e58-a620-3c75ce3687cd'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:12:18.867972 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a0dc5bdd-d9ef-4e58-a620-3c75ce3687cd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a4f90e9a8c164448a214481704ff18a2', 'request_id': 'req-a0dc5bdd-d9ef-4e58-a620-3c75ce3687cd'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:12:18.875143 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a0dc5bdd-d9ef-4e58-a620-3c75ce3687cd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a4f90e9a8c164448a214481704ff18a2', 'request_id': 'req-a0dc5bdd-d9ef-4e58-a620-3c75ce3687cd'} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:12:18.886280 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a0dc5bdd-d9ef-4e58-a620-3c75ce3687cd admin admin] The token cache is being invalidate because project a4f90e9a8c164448a214481704ff18a2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81529) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 18 01:12:18.886589 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a0dc5bdd-d9ef-4e58-a620-3c75ce3687cd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:12:18.887202 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG keystone.notifications [None req-a0dc5bdd-d9ef-4e58-a620-3c75ce3687cd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81529) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 18 01:12:18.888641 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: [pid: 81529|app: 0|req: 1470/2931] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 18 01:12:18 2022] DELETE /identity/v3/projects/a4f90e9a8c164448a214481704ff18a2 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 18 01:12:28.566285 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=29,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:12:28.882912 dv-cinder-master-3-iscsi devstack@keystone.service[81529]: DEBUG dbcounter [-] [81529] Writing DB stats keystone:SELECT=57,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81529) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 18 01:16:06.980852 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89304bfb-eeae-449a-ab50-0dd072cd326d admin admin] Authenticating user token {{(pid=81530) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 18 01:16:06.982523 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-89304bfb-eeae-449a-ab50-0dd072cd326d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:16:07.122345 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89304bfb-eeae-449a-ab50-0dd072cd326d admin admin] Validating token access rules against request {{(pid=81530) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 18 01:16:07.182214 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a378b584-64e7-410a-9ed9-f46579b3a705 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8f30116fb5ce43dab69efa8bafd9d049', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c520a8da3dc54adba4f7cc04ef42e367', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81530) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 18 01:16:07.182848 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a378b584-64e7-410a-9ed9-f46579b3a705 service placement] REQUEST_METHOD: `GET` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 18 01:16:07.183127 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a378b584-64e7-410a-9ed9-f46579b3a705 service placement] SCRIPT_NAME: `/identity` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 18 01:16:07.183397 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a378b584-64e7-410a-9ed9-f46579b3a705 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81530) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 18 01:16:07.185116 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.fernet_utils [None req-a378b584-64e7-410a-9ed9-f46579b3a705 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81530) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 18 01:16:07.195861 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a378b584-64e7-410a-9ed9-f46579b3a705 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 18 01:16:07.197415 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a378b584-64e7-410a-9ed9-f46579b3a705 service placement] RBAC: Authorization granted {{(pid=81530) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 18 01:16:07.207526 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: [pid: 81530|app: 0|req: 1462/2932] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 18 01:16:06 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 227 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 18 01:16:17.202832 dv-cinder-master-3-iscsi devstack@keystone.service[81530]: DEBUG dbcounter [-] [81530] Writing DB stats keystone:SELECT=89 {{(pid=81530) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}